[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489068667 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002342] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006799] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009132] LSM: Security Framework initializing [ 0.010049] Yama: becoming mindful. [ 0.011033] SELinux: Initializing. [ 0.012063] *** VALIDATE selinux *** [ 0.019652] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023618] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027118] *** VALIDATE tmpfs *** [ 0.028221] *** VALIDATE proc *** [ 0.029085] *** VALIDATE cgroup *** [ 0.029780] *** VALIDATE cgroup2 *** [ 0.030258] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031118] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033027] Spectre V2 : User space: Vulnerable [ 0.034007] Speculative Store Bypass: Vulnerable [ 0.036914] debug: unmapping init [mem 0xffffffff93259000-0xffffffff93260fff] [ 0.039000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039737] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040020] ... version: 2 [ 0.040954] ... bit width: 48 [ 0.041018] ... generic registers: 4 [ 0.042010] ... value mask: 0000ffffffffffff [ 0.043013] ... max period: 00007fffffffffff [ 0.044008] ... fixed-purpose events: 3 [ 0.045008] ... event mask: 000000070000000f [ 0.046312] rcu: Hierarchical SRCU implementation. [ 0.048360] smp: Bringing up secondary CPUs ... [ 0.049611] x86: Booting SMP configuration: [ 0.050032] .... node #0, CPUs: #1 #2 #3 [ 0.054014] smp: Brought up 1 node, 4 CPUs [ 0.056016] smpboot: Max logical packages: 1 [ 0.057009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137093] node 0 deferred pages initialised in 77ms [ 0.139200] devtmpfs: initialized [ 0.140361] x86/mm: Memory block size: 128MB [ 0.143688] gcov: version magic: 0x41383552 [ 0.144811] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148060] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149234] pinctrl core: initialized pinctrl subsystem [ 0.150205] [ 0.150539] ************************************************************* [ 0.151016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152012] ** ** [ 0.153011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154013] ** ** [ 0.155013] ** This means that this kernel is built to expose internal ** [ 0.156011] ** IOMMU data structures, which may compromise security on ** [ 0.157014] ** your system. ** [ 0.158012] ** ** [ 0.159010] ** If you see this message and you are not debugging the ** [ 0.160011] ** kernel, report this immediately to your vendor! ** [ 0.161011] ** ** [ 0.162013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163010] ************************************************************* [ 0.164768] NET: Registered protocol family 16 [ 0.165424] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.167067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.168496] cpuidle: using governor menu [ 0.170685] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.172598] PCI: Using configuration type 1 for base access [ 0.174113] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.182051] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.183012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.185069] cryptd: max_cpu_qlen set to 1000 [ 0.187081] ACPI: Added _OSI(Module Device) [ 0.188010] ACPI: Added _OSI(Processor Device) [ 0.189013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.190013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.193292] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.195507] ACPI: Interpreter enabled [ 0.196088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.197019] ACPI: Using IOAPIC for interrupt routing [ 0.198116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.199474] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.208628] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.209052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.210038] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.211097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.213065] acpiphp: Slot [2] registered [ 0.214043] acpiphp: Slot [5] registered [ 0.215027] acpiphp: Slot [6] registered [ 0.216010] acpiphp: Slot [3] registered [ 0.216944] acpiphp: Slot [4] registered [ 0.217066] acpiphp: Slot [7] registered [ 0.218105] acpiphp: Slot [8] registered [ 0.219044] acpiphp: Slot [9] registered [ 0.219967] acpiphp: Slot [10] registered [ 0.220079] acpiphp: Slot [11] registered [ 0.220942] acpiphp: Slot [12] registered [ 0.221059] acpiphp: Slot [13] registered [ 0.222057] acpiphp: Slot [14] registered [ 0.223084] acpiphp: Slot [15] registered [ 0.224056] acpiphp: Slot [16] registered [ 0.225061] acpiphp: Slot [17] registered [ 0.226030] acpiphp: Slot [18] registered [ 0.227040] acpiphp: Slot [19] registered [ 0.228000] acpiphp: Slot [20] registered [ 0.229000] acpiphp: Slot [21] registered [ 0.230000] acpiphp: Slot [22] registered [ 0.230991] acpiphp: Slot [23] registered [ 0.231076] acpiphp: Slot [24] registered [ 0.231983] acpiphp: Slot [25] registered [ 0.232071] acpiphp: Slot [26] registered [ 0.233083] acpiphp: Slot [27] registered [ 0.234088] acpiphp: Slot [28] registered [ 0.235080] acpiphp: Slot [29] registered [ 0.236025] acpiphp: Slot [30] registered [ 0.236953] acpiphp: Slot [31] registered [ 0.237069] PCI host bridge to bus 0000:00 [ 0.238024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.239018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.240019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.241028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.242026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.243023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.244172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.245898] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.247046] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.251319] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.253765] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.254024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.255013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.256020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.257519] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.258565] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.259037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.260698] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.262632] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.267017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.269022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.272375] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.275013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.278021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.285032] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.290909] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.293018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.296018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.302017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.308602] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.309288] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.310275] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.311233] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.312169] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.315508] iommu: Default domain type: Passthrough [ 0.316947] SCSI subsystem initialized [ 0.317112] ACPI: bus type USB registered [ 0.318081] usbcore: registered new interface driver usbfs [ 0.319056] usbcore: registered new interface driver hub [ 0.320048] usbcore: registered new device driver usb [ 0.321115] pps_core: LinuxPPS API ver. 1 registered [ 0.322007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.323069] PTP clock support registered [ 0.324198] EDAC MC: Ver: 3.0.0 [ 0.325331] PCI: Using ACPI for IRQ routing [ 0.326600] NetLabel: Initializing [ 0.327007] NetLabel: domain hash size = 128 [ 0.328010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.329068] NetLabel: unlabeled traffic allowed by default [ 0.330115] vgaarb: loaded [ 0.331210] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.332008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.340143] clocksource: Switched to clocksource kvm-clock [ 0.424565] VFS: Disk quotas dquot_6.6.0 [ 0.425566] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.427240] *** VALIDATE ramfs *** [ 0.428035] *** VALIDATE hugetlbfs *** [ 0.429042] pnp: PnP ACPI init [ 0.430791] pnp: PnP ACPI: found 6 devices [ 0.446776] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.448930] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.450243] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.451531] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.452979] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.454428] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.456164] NET: Registered protocol family 2 [ 0.457843] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.461117] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.463459] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.467337] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.469711] TCP: Hash tables configured (established 65536 bind 65536) [ 0.471629] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.473652] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.475438] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.477372] NET: Registered protocol family 1 [ 0.479281] RPC: Registered named UNIX socket transport module. [ 0.480666] RPC: Registered udp transport module. [ 0.481757] RPC: Registered tcp transport module. [ 0.482821] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.484294] NET: Registered protocol family 44 [ 0.485749] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.487262] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.488471] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.489842] PCI: CLS 0 bytes, default 64 [ 0.490993] Unpacking initramfs... [ 1.818747] debug: unmapping init [mem 0xffff9fe5bcc64000-0xffff9fe5bffcffff] [ 1.821830] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.823437] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.825549] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.237628] Initialise system trusted keyrings [ 2.238766] Key type blacklist registered [ 2.239901] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.246249] zbud: loaded [ 2.248564] *** VALIDATE nfs *** [ 2.249282] *** VALIDATE nfs4 *** [ 2.250214] pstore: using deflate compression [ 2.252323] Platform Keyring initialized [ 2.338564] NET: Registered protocol family 38 [ 2.339965] Key type asymmetric registered [ 2.340736] Asymmetric key parser 'x509' registered [ 2.341837] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.343995] io scheduler mq-deadline registered [ 2.345261] io scheduler kyber registered [ 2.346288] io scheduler bfq registered [ 2.347623] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.349680] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.351566] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.353402] ACPI: Power Button [PWRF] [ 2.356846] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.361794] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.368402] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.394777] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.421177] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.424336] Non-volatile memory driver v1.3 [ 2.425330] Linux agpgart interface v0.103 [ 2.448942] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.450667] vda: detected capacity change from 0 to 74702848 [ 2.461159] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.463058] vdb: detected capacity change from 0 to 1073741824 [ 2.467319] libphy: Fixed MDIO Bus: probed [ 2.471711] usbcore: registered new interface driver usbserial_generic [ 2.473281] usbserial: USB Serial support registered for generic [ 2.474638] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.477467] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.478526] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.480018] mousedev: PS/2 mouse device common for all mice [ 2.481867] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.484079] rtc_cmos 00:05: RTC can wake from S4 [ 2.486179] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.489443] rtc_cmos 00:05: registered as rtc0 [ 2.491973] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.492197] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.496090] intel_pstate: CPU model not supported [ 2.498094] hid: raw HID events driver (C) Jiri Kosina [ 2.500291] usbcore: registered new interface driver usbhid [ 2.502291] usbhid: USB HID core driver [ 2.503392] drop_monitor: Initializing network drop monitor service [ 2.505593] Initializing XFRM netlink socket [ 2.506968] NET: Registered protocol family 10 [ 2.509133] Segment Routing with IPv6 [ 2.510031] NET: Registered protocol family 17 [ 2.511320] mpls_gso: MPLS GSO support [ 2.515219] RAS: Correctable Errors collector initialized. [ 2.516377] AVX version of gcm_enc/dec engaged. [ 2.517271] AES CTR mode by8 optimization enabled [ 2.573444] sched_clock: Marking stable (2573428185, 0)->(3420551848, -847123663) [ 2.576252] registered taskstats version 1 [ 2.577760] Loading compiled-in X.509 certificates [ 2.579212] zswap: loaded using pool lzo/zbud [ 2.600272] Key type big_key registered [ 2.609824] Key type encrypted registered [ 2.611098] ima: No TPM chip found, activating TPM-bypass! [ 2.612512] ima: Allocated hash algorithm: sha1 [ 2.613846] ima: No architecture policies found [ 2.614933] evm: Initialising EVM extended attributes: [ 2.615981] evm: security.selinux [ 2.616642] evm: security.ima [ 2.617401] evm: security.capability [ 2.618423] evm: HMAC attrs: 0x1 [ 2.620181] rtc_cmos 00:05: setting system clock to 2026-08-15 08:53:11 UTC (1786783991) [ 2.624640] debug: unmapping init [mem 0xffffffff94203000-0xffffffff943fffff] [ 2.626594] debug: unmapping init [mem 0xffffffff92f82000-0xffffffff93258fff] [ 2.634070] Write protecting the kernel read-only data: 28672k [ 2.636268] debug: unmapping init [mem 0xffffffff91603000-0xffffffff917fffff] [ 2.638144] debug: unmapping init [mem 0xffffffff91f14000-0xffffffff91ffffff] [ 2.659879] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.665339] systemd[1]: Detected virtualization kvm. [ 2.667088] systemd[1]: Detected architecture x86-64. [ 2.669394] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.693381] systemd[1]: No hostname configured. [ 2.694470] systemd[1]: Set hostname to . [ 2.695659] random: systemd: uninitialized urandom read (16 bytes read) [ 2.697094] systemd[1]: Initializing machine ID from random generator. [ 2.744758] random: ln: uninitialized urandom read (6 bytes read) [ 2.822830] random: systemd: uninitialized urandom read (16 bytes read) [ 2.824467] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 2.827362] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 2.831072] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.402400] device-mapper: uevent: version 1.0.3 [ 3.405387] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.170374] virtio_net virtio0 ens2: renamed from eth0 [ 4.174600] random: fast init done [ 4.323478] scsi host0: ata_piix [ 4.371807] scsi host1: ata_piix [ 4.373216] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.375222] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.029662] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.125479] random: crng init done [ 9.126955] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.559061] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.634825] printk: systemd: 24 output lines suppressed due to ratelimiting [ 10.869941] SELinux: Disabled at runtime. [ 10.932149] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.941400] systemd[1]: Detected virtualization kvm. [ 10.943497] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.429691] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.433370] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.437898] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.441934] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.449299] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.455542] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.462699] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Mounting Kernel Debug File System... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ 11.620477] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 11.927903] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.256861] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.291730] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.387784] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.403575] EDAC sbridge: Ver: 1.1.2 [ 13.768406] Key type dns_resolver registered [ 14.064483] NFS: Registering the id_resolver key type [ 14.066480] Key type id_resolver registered [ 14.068072] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg132-client login: [ 41.161833] libcfs: loading out-of-tree module taints kernel. [ 41.240773] Key type ._llcrypt registered [ 41.242425] Key type .llcrypt registered [ 41.462384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 41.470060] alg: No test for adler32 (adler32-zlib) [ 42.480919] Lustre: Lustre: Build Version: 2.17.57_1_g05abe65 [ 42.806268] LNet: Added LNI 192.168.201.32@tcp [8/256/0/180] [ 44.423892] Key type lgssc registered [ 46.122757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 133.892374] hrtimer: interrupt took 4364092 ns [ 211.755737] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 217.428754] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 228.917958] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing check_logdir /tmp/testlogs/ [ 236.666481] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing yml_node [ 237.540235] Lustre: lustre-OST0000-osc-ffff9fe609704000: disconnect after 23s idle [ 241.655254] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 244.704891] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 247.109868] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 248.896709] Lustre: DEBUG MARKER: -----============= acceptance-small: recovery-small ============----- Sat Aug 15 04:57:16 EDT 2026 [ 270.036161] Lustre: DEBUG MARKER: excepting tests: 136 [ 272.270679] Lustre: DEBUG MARKER: === recovery-small: start setup 04:57:39 (1786784259) === [ 282.011502] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing check_config_client /mnt/lustre [ 302.755416] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 315.862500] Lustre: DEBUG MARKER: === recovery-small: finish setup 04:58:23 (1786784303) === [ 317.542380] Lustre: DEBUG MARKER: == recovery-small test 1: create, chmod, stat: drop req, drop rep ========================================================== 04:58:25 (1786784305) [ 334.304430] Lustre: 10823:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786784307/real 1786784307] req@ffff9fe6130cce00 x1873578852105728/t0(0) o700->lustre-MDT0000-mdc-ffff9fe609704000@192.168.201.132@tcp:30/10 lens 264/248 e 0 to 1 dl 1786784323 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 334.347846] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 334.401123] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp) [ 353.759761] Lustre: 10844:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786784326/real 1786784326] req@ffff9fe615aa2a00 x1873578852108032/t0(0) o36->lustre-MDT0000-mdc-ffff9fe609704000@192.168.201.132@tcp:12/10 lens 520/576 e 0 to 1 dl 1786784342 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 353.775071] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 353.805355] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp) [ 372.703148] Lustre: 10870:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786784345/real 1786784345] req@ffff9fe609644700 x1873578852109824/t0(0) o101->lustre-MDT0000-mdc-ffff9fe609704000@192.168.201.132@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786784361 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:0 [ 372.717567] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 372.756557] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp) [ 391.647348] Lustre: 10891:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786784364/real 1786784364] req@ffff9fe609644000 x1873578852112640/t0(0) o36->lustre-MDT0000-mdc-ffff9fe609704000@192.168.201.132@tcp:12/10 lens 488/512 e 0 to 1 dl 1786784380 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:4294967295 [ 391.704441] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 391.773821] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp) [ 411.615210] Lustre: 10917:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786784384/real 1786784384] req@ffff9fe615aa1c00 x1873578852114432/t0(0) o34->lustre-MDT0000-mdc-ffff9fe609704000@192.168.201.132@tcp:12/10 lens 472/728 e 0 to 1 dl 1786784400 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 411.652546] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 411.698590] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp) [ 430.047431] Lustre: 10938:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786784402/real 1786784402] req@ffff9fe6130cc000 x1873578852116224/t0(0) o34->lustre-MDT0000-mdc-ffff9fe609704000@192.168.201.132@tcp:12/10 lens 472/728 e 0 to 1 dl 1786784418 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 430.058292] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 430.090964] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp) [ 437.418693] Lustre: DEBUG MARKER: == recovery-small test 4: open: drop req, drop rep ======= 05:00:24 (1786784424) [ 455.135280] Lustre: 11550:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786784427/real 1786784427] req@ffff9fe609647800 x1873578852119168/t0(0) o101->lustre-MDT0000-mdc-ffff9fe609704000@192.168.201.132@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786784443 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 [ 455.158201] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 455.235501] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp) [ 481.694832] Lustre: DEBUG MARKER: == recovery-small test 5: rename: drop req, drop rep ===== 05:01:09 (1786784469) [ 498.143116] Lustre: 12192:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786784471/real 1786784471] req@ffff9fe615aa3100 x1873578852126464/t0(0) o101->lustre-MDT0000-mdc-ffff9fe609704000@192.168.201.132@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786784487 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'mv.0' uid:0 gid:0 projid:0 [ 498.155419] Lustre: 12192:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 498.158591] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 498.165641] Lustre: Skipped 1 previous similar message [ 498.233067] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp) [ 498.258561] Lustre: Skipped 1 previous similar message [ 525.572164] Lustre: DEBUG MARKER: == recovery-small test 6: link, unlink: drop req, drop rep ========================================================== 05:01:53 (1786784513) [ 547.809855] Lustre: lustre-OST0001-osc-ffff9fe609704000: disconnect after 23s idle [ 547.816797] Lustre: Skipped 1 previous similar message [ 580.575322] Lustre: 12880:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786784553/real 1786784553] req@ffff9fe60966b100 x1873578852141312/t0(0) o101->lustre-MDT0000-mdc-ffff9fe609704000@192.168.201.132@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786784569 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'unlink.0' uid:0 gid:0 projid:0 [ 580.608180] Lustre: 12880:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 580.619400] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 580.648276] Lustre: Skipped 3 previous similar messages [ 580.681476] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp) [ 580.690075] Lustre: Skipped 3 previous similar messages [ 608.046548] Lustre: DEBUG MARKER: == recovery-small test 8: touch: drop rep (bug 1423) ===== 05:03:15 (1786784595) [ 634.456601] Lustre: DEBUG MARKER: == recovery-small test 9: pause bulk on OST (bug 1420) === 05:03:41 (1786784621) [ 648.701182] Lustre: DEBUG MARKER: == recovery-small test 10a: finish request on server after client eviction (bug 1521) ========================================================== 05:03:56 (1786784636) [ 649.073664] Lustre: *** cfs_fail_loc=305, val=0*** [ 665.656765] Lustre: *** cfs_fail_loc=305, val=0*** [ 672.224480] Lustre: lustre-OST0001-osc-ffff9fe609704000: disconnect after 24s idle [ 681.014613] Lustre: *** cfs_fail_loc=305, val=0*** [ 697.385815] Lustre: *** cfs_fail_loc=305, val=0*** [ 713.782421] Lustre: *** cfs_fail_loc=305, val=0*** [ 729.126579] Lustre: *** cfs_fail_loc=305, val=0*** [ 745.522610] Lustre: *** cfs_fail_loc=305, val=0*** [ 762.177680] LustreError: lustre-MDT0000-mdc-ffff9fe609704000: operation ldlm_enqueue to node 192.168.201.132@tcp failed: rc = -107 [ 762.183762] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 762.193288] Lustre: Skipped 2 previous similar messages [ 762.202052] LustreError: lustre-MDT0000-mdc-ffff9fe609704000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 762.214870] LustreError: 14841:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9fe609704000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9fe610214e00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 762.229396] Lustre: lustre-MDT0000-mdc-ffff9fe609704000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp) [ 762.235215] Lustre: Skipped 2 previous similar messages [ 771.049464] Lustre: DEBUG MARKER: == recovery-small test 10b: re-send BL AST =============== 05:05:58 (1786784758) [ 795.561159] Lustre: DEBUG MARKER: == recovery-small test 10c: re-send BL AST vs reconnect race (LU-5569) ========================================================== 05:06:23 (1786784783) [ 796.344305] Lustre: *** cfs_fail_loc=305, val=0*** [ 796.346183] Lustre: Skipped 1 previous similar message [ 802.806674] Lustre: DEBUG MARKER: == recovery-small test 10d: test failed blocking ast ===== 05:06:30 (1786784790) [ 806.127172] Lustre: Unmounted lustre-client [ 806.608599] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 808.147751] LustreError: lustre-OST0000-osc-ffff9fe615ac2000: operation ldlm_enqueue to node 192.168.201.132@tcp failed: rc = -107 [ 808.159655] LustreError: lustre-OST0000-osc-ffff9fe615ac2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 808.178720] Lustre: 2392:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.132@tcp:/lustre/fid: [0x200000404:0x1:0x0]/ may get corrupted (rc -108) [ 811.209683] Lustre: Unmounted lustre-client [ 819.112922] Lustre: DEBUG MARKER: == recovery-small test 10e: re-send BL AST vs reconnect race 2 ========================================================== 05:06:46 (1786784806) [ 821.809631] Lustre: DEBUG MARKER: SKIP: recovery-small test_10e need two clients [ 823.364919] Lustre: DEBUG MARKER: == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) ========================================================== 05:06:51 (1786784811) [ 848.952549] Lustre: DEBUG MARKER: == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) ========================================================== 05:07:16 (1786784836) [ 849.373826] Lustre: DEBUG MARKER: multiop /mnt/lustre/f12.recovery-small OS_c [ 866.783971] Lustre: 18559:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786784839/real 1786784839] req@ffff9fe609669c00 x1873578852221568/t0(0) o35->lustre-MDT0000-mdc-ffff9fe615ac2000@192.168.201.132@tcp:23/10 lens 392/624 e 0 to 1 dl 1786784855 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 [ 866.832110] Lustre: 18559:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 893.938273] Lustre: DEBUG MARKER: == recovery-small test 13: mdc_readpage restart test (bug 1138) ========================================================== 05:08:01 (1786784881) [ 919.406809] Lustre: DEBUG MARKER: == recovery-small test 14: mdc_readpage resend test (bug 1138) ========================================================== 05:08:27 (1786784907) [ 928.252767] Lustre: DEBUG MARKER: == recovery-small test 15: failed open (-ENOMEM) ========= 05:08:35 (1786784915) [ 934.601177] Lustre: DEBUG MARKER: == recovery-small test 16: timeout bulk put, don't evict client (2732) ========================================================== 05:08:42 (1786784922) [ 981.285283] Lustre: DEBUG MARKER: == recovery-small test 17a: timeout bulk get, don't evict client (2732) ========================================================== 05:09:28 (1786784968) [ 1041.277952] Lustre: DEBUG MARKER: == recovery-small test 17b: timeout bulk get, dont evict client (3582) ========================================================== 05:10:28 (1786785028) [ 1043.694672] Lustre: DEBUG MARKER: SKIP: recovery-small test_17b Needs multiple clients [ 1046.408976] Lustre: DEBUG MARKER: == recovery-small test 18a: manual ost invalidate clears page cache immediately ========================================================== 05:10:33 (1786785033) [ 1047.556648] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1047.624309] Lustre: lustre-OST0001-osc-ffff9fe615ac2000: Connection to lustre-OST0001 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1047.632437] Lustre: Skipped 5 previous similar messages [ 1047.647157] LustreError: lustre-OST0001-osc-ffff9fe615ac2000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1047.658465] Lustre: lustre-OST0001-osc-ffff9fe615ac2000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp) [ 1047.672670] Lustre: Skipped 4 previous similar messages [ 1057.417902] Lustre: DEBUG MARKER: == recovery-small test 18b: eviction and reconnect clears page cache (2766) ========================================================== 05:10:44 (1786785044) [ 1064.470130] LustreError: lustre-OST0000-osc-ffff9fe615ac2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1091.168536] Lustre: DEBUG MARKER: == recovery-small test 18c: Dropped connect reply after eviction handing (14755) ========================================================== 05:11:18 (1786785078) [ 1105.422157] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.201.132@tcp) after server handle changed from 0x6e753b230b7d3a8b to 0x6e753b230b7d3bf0 [ 1105.442592] LustreError: lustre-OST0000-osc-ffff9fe615ac2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1115.285040] Lustre: DEBUG MARKER: == recovery-small test 19a: test expired_lock_main on mds (2867) ========================================================== 05:11:42 (1786785102) [ 1115.882921] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1115.886378] Lustre: Skipped 1 previous similar message [ 1133.535166] Lustre: 2389:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786785106/real 1786785106] req@ffff9fe609645f80 x1873578852289280/t0(0) o103->lustre-MDT0000-mdc-ffff9fe615ac2000@192.168.201.132@tcp:17/18 lens 328/224 e 0 to 1 dl 1786785122 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 [ 1133.567447] Lustre: 2389:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1220.599836] LustreError: lustre-MDT0000-mdc-ffff9fe615ac2000: operation ldlm_enqueue to node 192.168.201.132@tcp failed: rc = -107 [ 1220.640245] LustreError: lustre-MDT0000-mdc-ffff9fe615ac2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1220.677341] LustreError: 24887:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5 [ 1221.649786] Lustre: Unmounted lustre-client [ 1228.359138] Lustre: DEBUG MARKER: == recovery-small test 19b: test expired_lock_main on ost (2867) ========================================================== 05:13:35 (1786785215) [ 1229.089478] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1336.835317] LustreError: lustre-OST0000-osc-ffff9fe615ac2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1337.753206] Lustre: Unmounted lustre-client [ 1345.547668] Lustre: DEBUG MARKER: == recovery-small test 19c: check reconnect and lock resend do not trigger expired_lock_main ========================================================== 05:15:33 (1786785333) [ 1346.658260] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1353.591982] Lustre: Unmounted lustre-client [ 1366.786287] Lustre: DEBUG MARKER: == recovery-small test 20a: ldlm_handle_enqueue error (should return error) ========================================================== 05:15:54 (1786785354) [ 1368.990329] LustreError: lustre-OST0000-osc-ffff9fe615ac2000: operation ldlm_enqueue to node 192.168.201.132@tcp failed: rc = -12 [ 1375.602856] Lustre: DEBUG MARKER: == recovery-small test 20b: ldlm_handle_enqueue error (should return error) ========================================================== 05:16:02 (1786785362) [ 1377.137433] LustreError: lustre-OST0000-osc-ffff9fe615ac2000: operation ldlm_enqueue to node 192.168.201.132@tcp failed: rc = -12 [ 1384.874381] Lustre: DEBUG MARKER: == recovery-small test 21a: drop close request while close and open are both in flight ========================================================== 05:16:12 (1786785372) [ 1413.523430] Lustre: DEBUG MARKER: == recovery-small test 21b: drop open request while close and open are both in flight ========================================================== 05:16:41 (1786785401) [ 1558.739969] Lustre: DEBUG MARKER: == recovery-small test 21c: drop both request while close and open are both in flight ========================================================== 05:19:05 (1786785545) [ 1581.535753] Lustre: lustre-MDT0000-mdc-ffff9fe615ac2000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1581.556665] Lustre: Skipped 18 previous similar messages [ 1581.574700] Lustre: lustre-MDT0000-mdc-ffff9fe615ac2000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp) [ 1581.592720] Lustre: Skipped 16 previous similar messages [ 1589.847385] Lustre: DEBUG MARKER: == recovery-small test 21d: drop close reply while close and open are both in flight ========================================================== 05:19:37 (1786785577) [ 1620.804134] Lustre: DEBUG MARKER: == recovery-small test 21e: drop open reply while close and open are both in flight ========================================================== 05:20:08 (1786785608) [ 1758.175167] Lustre: 30730:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786785611/real 1786785611] req@ffff9fe615aa1f80 x1873578852422400/t0(0) o36->lustre-MDT0000-mdc-ffff9fe615ac2000@192.168.201.132@tcp:12/10 lens 488/512 e 0 to 1 dl 1786785747 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 1758.250772] Lustre: 30730:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 1760.845592] Lustre: DEBUG MARKER: == recovery-small test 21f: drop both reply while close and open are both in flight ========================================================== 05:22:27 (1786785747) [ 1790.502406] Lustre: DEBUG MARKER: == recovery-small test 21g: drop open reply and close request while close and open are both in flight ========================================================== 05:22:57 (1786785777) [ 1819.431862] Lustre: DEBUG MARKER: == recovery-small test 21h: drop open request and close reply while close and open are both in flight ========================================================== 05:23:26 (1786785806) [ 1848.635230] Lustre: DEBUG MARKER: == recovery-small test 22: drop close request and do mknod ========================================================== 05:23:55 (1786785835) [ 1872.795978] Lustre: DEBUG MARKER: == recovery-small test 23: client hang when close a file after mds crash ========================================================== 05:24:20 (1786785860) [ 1900.383529] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail [ 1910.792307] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0x6e753b230b7d30dd to 0x6e753b230b7d595c [ 1925.052336] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1926.520223] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1936.344525] Lustre: DEBUG MARKER: == recovery-small test 24a: fsync error (should return error) ========================================================== 05:25:23 (1786785923) [ 1938.668153] LustreError: lustre-OST0000-osc-ffff9fe615ac2000: operation ost_write to node 192.168.201.132@tcp failed: rc = -107 [ 1938.703960] LustreError: lustre-OST0000-osc-ffff9fe615ac2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1938.730545] Lustre: 2390:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.132@tcp:/lustre/fid: [0x240000402:0x8:0x0]// may get corrupted (rc -5) [ 1950.696955] Lustre: DEBUG MARKER: == recovery-small test 24b: test dirty page discard due to client eviction ========================================================== 05:25:37 (1786785937) [ 1953.556401] LustreError: lustre-OST0000-osc-ffff9fe615ac2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1957.144076] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0 [ 1962.760210] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ FAIL: no discarded dirty page found! [ 1976.937636] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 05:26:03 (1786785963) [ 1979.864542] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node [ 1981.855742] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 05:26:09 (1786785969) [ 1984.050264] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node [ 1985.715992] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 05:26:13 (1786785973) [ 1989.636889] LustreError: lustre-MDT0000-mdc-ffff9fe615ac2000: operation mds_close to node 192.168.201.132@tcp failed: rc = -19 [ 1989.656629] LustreError: Skipped 2 previous similar messages [ 2009.060451] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail [ 2019.331971] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0x6e753b230b7d595c to 0x6e753b230b7d7658 [ 2019.379191] BUG: spinlock recursion on CPU#0, ptlrpcd_rcv/2388 [ 2019.386678] lock: 0xffff9fe615422ae0, .magic: dead4ead, .owner: ptlrpcd_rcv/2388, .owner_cpu: 0 [ 2019.395939] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2019.412471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2019.426429] Call Trace: [ 2019.428422] ? dump_stack+0xbb/0x10e [ 2019.432061] ? spin_dump.cold.3+0x32/0x65 [ 2019.435843] ? do_raw_spin_lock+0x10f/0x190 [ 2019.442403] ? _raw_spin_lock+0x21/0x30 [ 2019.449721] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2019.462799] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2019.473634] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2019.481778] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2019.488621] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2019.497843] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2019.506865] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2019.512066] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2019.515366] ? do_wait_intr+0xf0/0xf0 [ 2019.518562] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2019.522904] ? kthread+0x1d1/0x200 [ 2019.525332] ? set_kthread_struct+0x70/0x70 [ 2019.535523] ? ret_from_fork+0x1f/0x30 [ 2044.049461] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2044.052027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2044.054046] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2044.054058] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2044.054156] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2044.054162] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2044.054164] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2044.054177] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2044.054181] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2044.054186] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2044.054188] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2044.054196] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2044.054198] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2044.054201] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2044.054204] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2044.054207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2044.054209] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2044.054216] Call Trace: [ 2044.054222] [ 2044.054227] ? show_regs.cold.9+0x22/0x2f [ 2044.054236] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2044.054243] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2044.054247] ? watchdog+0x40/0x40 [ 2044.054251] ? hrtimer_interrupt+0x128/0x2c0 [ 2044.054256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2044.054262] ? apic_timer_interrupt+0xf/0x20 [ 2044.054267] [ 2044.054270] ? native_safe_halt+0x2e/0x50 [ 2044.054275] kvm_wait+0xcd/0xe0 [ 2044.054283] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2044.054292] do_raw_spin_lock+0x177/0x190 [ 2044.054296] _raw_spin_lock+0x21/0x30 [ 2044.054300] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2044.054583] ? do_raw_spin_unlock+0x75/0x190 [ 2044.054590] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2044.054897] ? do_wait_intr+0xf0/0xf0 [ 2044.054904] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2044.055458] mdc_close+0x53b/0xd70 [mdc] [ 2044.055495] lmv_close+0x18d/0x490 [lmv] [ 2044.055530] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2044.055646] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2044.056207] ll_md_close+0x274/0x800 [lustre] [ 2044.056303] ll_file_release+0x128/0xc50 [lustre] [ 2044.056401] __fput+0x10d/0x440 [ 2044.056410] ____fput+0x12/0x20 [ 2044.056415] task_work_run+0xc6/0x110 [ 2044.056424] exit_to_usermode_loop+0x1dd/0x1f0 [ 2044.056432] do_syscall_64+0x3d6/0x440 [ 2044.056437] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2044.056446] RIP: 0033:0x7fdc0afbfcd8 [ 2044.056454] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2044.056458] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2044.056465] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2044.056467] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2044.056470] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2044.056757] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2044.056760] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2044.061601] Modules linked in: lustre(O) [ 2044.063592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2044.063670] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2044.063677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2044.063680] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2044.063694] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2044.063698] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2044.063702] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2044.063704] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2044.063707] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2044.063709] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2044.063711] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2044.063715] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2044.063717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2044.063720] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2044.063730] Call Trace: [ 2044.063734] [ 2044.063739] ? show_regs.cold.9+0x22/0x2f [ 2044.063749] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2044.063756] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2044.063761] ? watchdog+0x40/0x40 [ 2044.063765] ? hrtimer_interrupt+0x128/0x2c0 [ 2044.063769] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2044.063777] ? apic_timer_interrupt+0xf/0x20 [ 2044.063781] [ 2044.063784] ? native_safe_halt+0x2e/0x50 [ 2044.063788] kvm_wait+0xcd/0xe0 [ 2044.063795] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2044.063802] do_raw_spin_lock+0x177/0x190 [ 2044.063806] _raw_spin_lock+0x21/0x30 [ 2044.063810] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2044.064067] ? do_raw_spin_unlock+0x75/0x190 [ 2044.064262] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2044.064473] ? do_wait_intr+0xf0/0xf0 [ 2044.064482] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2044.064700] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2044.064920] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2044.065149] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2044.065271] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2044.065303] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2044.065331] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2044.065368] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2044.065457] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2044.069591] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2044.069670] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2044.069677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2044.069680] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2044.069694] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2044.069699] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2044.069703] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2044.069706] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2044.069708] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2044.069710] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2044.069713] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2044.069716] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2044.069719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2044.069721] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2044.069730] Call Trace: [ 2044.069736] [ 2044.069742] ? show_regs.cold.9+0x22/0x2f [ 2044.069752] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2044.069758] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2044.069762] ? watchdog+0x40/0x40 [ 2044.069767] ? hrtimer_interrupt+0x128/0x2c0 [ 2044.069772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2044.069779] ? apic_timer_interrupt+0xf/0x20 [ 2044.069784] [ 2044.069787] ? native_safe_halt+0x2e/0x50 [ 2044.069791] kvm_wait+0xcd/0xe0 [ 2044.069799] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2044.069805] ? dump_stack+0xf6/0x10e [ 2044.069810] do_raw_spin_lock+0x177/0x190 [ 2044.069814] _raw_spin_lock+0x21/0x30 [ 2044.069819] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2044.070112] obd_mod_free+0x46/0x140 [ptlrpc] [ 2044.070321] mdc_commit_open+0x91/0xb0 [mdc] [ 2044.070359] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2044.070552] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2044.070755] after_reply+0xe2c/0x1780 [ptlrpc] [ 2044.070959] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2044.094672] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2044.094718] lmv_intent_open+0x338/0x1150 [lmv] [ 2044.094749] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2044.094856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2044.094940] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2044.094963] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2044.095055] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2044.095236] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2044.095326] ? do_raw_spin_unlock+0x75/0x190 [ 2044.095340] ? _raw_spin_unlock+0x12/0x30 [ 2044.095347] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2044.095378] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2044.095399] ll_atomic_open+0x36a/0xd70 [lustre] [ 2044.095482] lookup_open+0x356/0xb40 [ 2044.095490] do_last+0x313/0xfc0 [ 2044.095495] ? nd_jump_root+0xe5/0x160 [ 2044.095499] ? path_init+0x437/0x520 [ 2044.095503] path_openat+0xf7/0x500 [ 2044.095508] ? do_raw_spin_unlock+0x75/0x190 [ 2044.095512] do_filp_open+0x99/0x140 [ 2044.095518] ? getname_flags+0x6e/0x330 [ 2044.095522] ? __check_object_size+0xff/0x256 [ 2044.095528] ? do_raw_spin_unlock+0x75/0x190 [ 2044.095532] ? _raw_spin_unlock+0x12/0x30 [ 2044.095536] do_sys_openat2+0x2b4/0x410 [ 2044.095540] do_sys_open+0x73/0xa0 [ 2044.095544] __x64_sys_openat+0x24/0x30 [ 2044.095547] do_syscall_64+0xc1/0x440 [ 2044.095554] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2044.095559] RIP: 0033:0x7fdc0afbf332 [ 2044.095567] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2044.095571] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2044.095576] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2044.095579] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2044.095581] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2044.095583] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2044.095585] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2044.513556] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2044.515448] ? do_wait_intr+0xf0/0xf0 [ 2044.516937] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2044.519238] kthread+0x1d1/0x200 [ 2044.520565] ? set_kthread_struct+0x70/0x70 [ 2044.522487] ret_from_fork+0x1f/0x30 [ 2048.052089] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2048.057175] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2048.100725] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2048.114604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2048.117645] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2048.129255] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2048.139146] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2048.170207] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2048.176677] RAX: 0000000000005888 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2048.189420] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2048.193772] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2048.199685] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2048.209995] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2048.219157] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2048.231857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2048.236142] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2048.247966] Call Trace: [ 2048.252223] [ 2048.252935] ? show_regs.cold.9+0x22/0x2f [ 2048.254232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2048.264444] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2048.271377] ? watchdog+0x40/0x40 [ 2048.276227] ? hrtimer_interrupt+0x128/0x2c0 [ 2048.281860] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2048.288782] ? apic_timer_interrupt+0xf/0x20 [ 2048.299199] [ 2048.300241] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2048.313824] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2048.324530] do_raw_spin_lock+0x177/0x190 [ 2048.331093] _raw_spin_lock+0x21/0x30 [ 2048.337563] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2048.348554] process_one_work+0x2c8/0x700 [ 2048.363782] worker_thread+0x296/0x6e0 [ 2048.364932] ? rescuer_thread+0x570/0x570 [ 2048.368955] kthread+0x1d1/0x200 [ 2048.370086] ? set_kthread_struct+0x70/0x70 [ 2048.371542] ret_from_fork+0x1f/0x30 [ 2059.231030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 34s! [ 2059.234688] Showing busy workqueues and worker pools: [ 2059.236870] workqueue events: flags=0x0 [ 2059.238191] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2059.241411] pending: cpu_latency_work [ptlrpc] [ 2059.244017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2059.246963] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2059.255125] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2059.268203] pending: kfree_rcu_monitor [ 2059.276712] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2059.285597] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2059.292456] workqueue events_long: flags=0x0 [ 2059.294781] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2059.301963] pending: lu_objects_flush [obdclass] [ 2059.305393] workqueue events_unbound: flags=0x2 [ 2059.311222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 2059.316817] pending: flush_memcg_stats_dwork [ 2059.320118] workqueue events_power_efficient: flags=0x80 [ 2059.323192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2059.326487] pending: neigh_periodic_work [ 2059.328961] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2059.332436] pending: do_cache_clean, neigh_periodic_work [ 2059.335179] workqueue mm_percpu_wq: flags=0x8 [ 2059.338264] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2059.341817] pending: lru_add_drain_per_cpu [ 2059.344238] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2059.350035] pending: vmstat_update, lru_add_drain_per_cpu [ 2059.355207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2059.363794] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2059.367126] workqueue writeback: flags=0x4a [ 2059.370157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2059.373473] pending: wb_workfn, wb_workfn [ 2059.375956] workqueue xprtiod: flags=0xa [ 2059.378189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2059.384976] pending: xs_stream_data_receive_workfn [ 2059.388627] workqueue knbd0-recv: flags=0x1a [ 2059.390750] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2059.393317] in-flight: 655:recv_work [ 2059.395200] workqueue ptlrpc_pinger: flags=0x2 [ 2059.397341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2059.400432] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2059.403249] pool 8: cpus=0-3 flags=0x4 nice=0 hung=34s workers=4 idle: 36396 38 11188 [ 2059.407176] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2049s workers=2 idle: 658 [ 2072.054433] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2072.058800] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2072.078105] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 2072.088609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2072.088699] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2072.088706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2072.088709] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2072.088723] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2072.088727] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2072.088732] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2072.088734] RDX: 000000000000769f RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2072.088737] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2072.088739] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2072.088742] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2072.088744] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2072.088748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2072.088750] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2072.088759] Call Trace: [ 2072.088764] [ 2072.088770] ? show_regs.cold.9+0x22/0x2f [ 2072.088779] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2072.088786] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2072.088790] ? watchdog+0x40/0x40 [ 2072.088795] ? hrtimer_interrupt+0x128/0x2c0 [ 2072.088799] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2072.088808] ? apic_timer_interrupt+0xf/0x20 [ 2072.088812] [ 2072.088815] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2072.088821] do_raw_spin_lock+0x177/0x190 [ 2072.088825] _raw_spin_lock+0x21/0x30 [ 2072.088832] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2072.089103] ? do_raw_spin_unlock+0x75/0x190 [ 2072.089108] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2072.127808] intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2072.127849] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2072.127856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2072.127859] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2072.127876] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2072.127880] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2072.127884] RAX: 0000000000007137 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2072.127886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2072.127889] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2072.127891] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2072.127893] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2072.127897] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2072.127900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2072.127902] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2072.127911] Call Trace: [ 2072.127916] [ 2072.127923] ? show_regs.cold.9+0x22/0x2f [ 2072.127933] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2072.127939] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2072.127944] ? watchdog+0x40/0x40 [ 2072.127948] ? hrtimer_interrupt+0x128/0x2c0 [ 2072.127951] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2072.127960] ? apic_timer_interrupt+0xf/0x20 [ 2072.127964] [ 2072.127967] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2072.127975] do_raw_spin_lock+0x177/0x190 [ 2072.127979] _raw_spin_lock+0x21/0x30 [ 2072.127983] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2072.129511] ? do_raw_spin_unlock+0x75/0x190 [ 2072.129519] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2072.129743] ? do_wait_intr+0xf0/0xf0 [ 2072.129750] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2072.129960] mdc_close+0x53b/0xd70 [mdc] [ 2072.130003] lmv_close+0x18d/0x490 [lmv] [ 2072.130034] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2072.130147] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2072.130227] ll_md_close+0x274/0x800 [lustre] [ 2072.130311] ll_file_release+0x128/0xc50 [lustre] [ 2072.131970] __fput+0x10d/0x440 [ 2072.131981] ____fput+0x12/0x20 [ 2072.131986] task_work_run+0xc6/0x110 [ 2072.132004] exit_to_usermode_loop+0x1dd/0x1f0 [ 2072.132011] do_syscall_64+0x3d6/0x440 [ 2072.132015] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2072.132024] RIP: 0033:0x7fdc0afbfcd8 [ 2072.132031] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2072.132035] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2072.132040] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2072.132043] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2072.132045] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2072.132047] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2072.132049] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2072.879384] ? do_wait_intr+0xf0/0xf0 [ 2072.886326] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2072.890548] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2072.897635] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2072.908912] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2072.918346] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2072.925387] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2072.929237] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2072.937480] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2072.949303] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2072.958432] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2072.965134] lmv_intent_open+0x338/0x1150 [lmv] [ 2072.968662] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2072.982791] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2072.992003] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2073.008254] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2073.014634] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2073.022673] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2073.030117] ? do_raw_spin_unlock+0x75/0x190 [ 2073.039832] ? _raw_spin_unlock+0x12/0x30 [ 2073.041503] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2073.051472] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2073.055977] ll_atomic_open+0x36a/0xd70 [lustre] [ 2073.059799] lookup_open+0x356/0xb40 [ 2073.064943] do_last+0x313/0xfc0 [ 2073.070810] ? nd_jump_root+0xe5/0x160 [ 2073.079212] ? path_init+0x437/0x520 [ 2073.084547] path_openat+0xf7/0x500 [ 2073.091806] ? do_raw_spin_unlock+0x75/0x190 [ 2073.100911] do_filp_open+0x99/0x140 [ 2073.104900] ? getname_flags+0x6e/0x330 [ 2073.115027] ? __check_object_size+0xff/0x256 [ 2073.118548] ? do_raw_spin_unlock+0x75/0x190 [ 2073.125325] ? _raw_spin_unlock+0x12/0x30 [ 2073.139082] do_sys_openat2+0x2b4/0x410 [ 2073.142152] do_sys_open+0x73/0xa0 [ 2073.150918] __x64_sys_openat+0x24/0x30 [ 2073.156333] do_syscall_64+0xc1/0x440 [ 2073.162460] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2073.169592] RIP: 0033:0x7fdc0afbf332 [ 2073.175571] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2073.206724] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2073.225066] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2073.243251] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2073.253777] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2073.261588] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2073.275710] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2076.051034] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2076.054220] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2076.120500] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2076.139205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2076.156380] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2076.168408] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2076.180439] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2076.201844] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2076.213299] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2076.226799] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2076.238207] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2076.243942] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2076.262039] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2076.275034] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2076.293649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2076.298788] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2076.313052] Call Trace: [ 2076.321011] [ 2076.322460] ? show_regs.cold.9+0x22/0x2f [ 2076.331105] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2076.336408] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2076.343869] ? watchdog+0x40/0x40 [ 2076.349965] ? hrtimer_interrupt+0x128/0x2c0 [ 2076.353909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2076.356444] ? apic_timer_interrupt+0xf/0x20 [ 2076.358374] [ 2076.362739] ? native_safe_halt+0x2e/0x50 [ 2076.368323] kvm_wait+0xcd/0xe0 [ 2076.369484] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2076.376062] do_raw_spin_lock+0x177/0x190 [ 2076.382429] _raw_spin_lock+0x21/0x30 [ 2076.384331] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2076.388139] process_one_work+0x2c8/0x700 [ 2076.390028] worker_thread+0x296/0x6e0 [ 2076.396903] ? rescuer_thread+0x570/0x570 [ 2076.400414] kthread+0x1d1/0x200 [ 2076.407249] ? set_kthread_struct+0x70/0x70 [ 2076.411021] ret_from_fork+0x1f/0x30 [ 2079.348258] rcu: INFO: rcu_sched self-detected stall on CPU [ 2079.351898] rcu: 0-...!: (37743 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 2079.366883] (t=60005 jiffies g=270761 q=2631) [ 2079.380246] rcu: rcu_sched kthread starved for 54937 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 2079.406641] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2079.434514] rcu: RCU grace-period kthread stack dump: [ 2079.439608] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2079.470604] Call Trace: [ 2079.475041] __schedule+0x351/0xcb0 [ 2079.482327] schedule+0xc0/0x180 [ 2079.483570] schedule_timeout+0xb4/0x190 [ 2079.495064] ? do_raw_spin_unlock+0x75/0x190 [ 2079.505493] ? __next_timer_interrupt+0x160/0x160 [ 2079.516355] ? prepare_to_swait_event+0xf3/0x140 [ 2079.538459] rcu_gp_kthread+0x8dd/0x15b0 [ 2079.544683] ? force_qs_rnp+0x2c0/0x2c0 [ 2079.558398] kthread+0x1d1/0x200 [ 2079.559609] ? set_kthread_struct+0x70/0x70 [ 2079.589605] ret_from_fork+0x1f/0x30 [ 2079.595983] rcu: Stack dump where RCU GP kthread last ran: [ 2079.616031] NMI backtrace for cpu 0 [ 2079.630916] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2079.664685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2079.691204] Call Trace: [ 2079.692069] [ 2079.692807] ? dump_stack+0xbb/0x10e [ 2079.713964] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2079.715766] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2079.734654] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2079.745651] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2079.760587] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2079.766917] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 2079.781288] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 2079.789235] ? update_process_times+0x6d/0xc0 [ 2079.795106] ? tick_sched_handle.isra.18+0x27/0x90 [ 2079.800024] ? tick_sched_timer+0x53/0xb0 [ 2079.809850] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2079.818592] ? tick_sched_handle.isra.18+0x90/0x90 [ 2079.825111] ? hrtimer_interrupt+0x128/0x2c0 [ 2079.831703] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2079.838918] ? apic_timer_interrupt+0xf/0x20 [ 2079.849636] [ 2079.851115] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2079.862418] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2079.878717] ? dump_stack+0xf6/0x10e [ 2079.888383] ? do_raw_spin_lock+0x177/0x190 [ 2079.896317] ? _raw_spin_lock+0x21/0x30 [ 2079.905714] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2079.919060] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2079.925925] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2079.931461] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2079.935239] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2079.952739] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2079.965953] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2079.985837] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2079.995722] ? do_wait_intr+0xf0/0xf0 [ 2080.003199] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2080.005152] ? kthread+0x1d1/0x200 [ 2080.024545] ? set_kthread_struct+0x70/0x70 [ 2080.025997] ? ret_from_fork+0x1f/0x30 [ 2080.043881] NMI backtrace for cpu 0 [ 2080.051331] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2080.067501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2080.081549] Call Trace: [ 2080.084998] [ 2080.086384] ? dump_stack+0xbb/0x10e [ 2080.091359] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2080.100514] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2080.103027] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2080.112578] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2080.117898] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2080.124045] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2080.127923] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2080.130990] ? update_process_times+0x6d/0xc0 [ 2080.135507] ? tick_sched_handle.isra.18+0x27/0x90 [ 2080.138005] ? tick_sched_timer+0x53/0xb0 [ 2080.143591] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2080.146545] ? tick_sched_handle.isra.18+0x90/0x90 [ 2080.150843] ? hrtimer_interrupt+0x128/0x2c0 [ 2080.156700] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2080.158953] ? apic_timer_interrupt+0xf/0x20 [ 2080.163018] [ 2080.164049] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2080.169197] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2080.181631] ? dump_stack+0xf6/0x10e [ 2080.186344] ? do_raw_spin_lock+0x177/0x190 [ 2080.191629] ? _raw_spin_lock+0x21/0x30 [ 2080.197579] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2080.209029] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2080.217316] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2080.218838] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2080.232561] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2080.242537] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2080.255285] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2080.261613] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2080.268781] ? do_wait_intr+0xf0/0xf0 [ 2080.274872] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2080.288189] ? kthread+0x1d1/0x200 [ 2080.291294] ? set_kthread_struct+0x70/0x70 [ 2080.295849] ? ret_from_fork+0x1f/0x30 [ 2089.439520] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 64s! [ 2089.442782] Showing busy workqueues and worker pools: [ 2089.448443] workqueue events: flags=0x0 [ 2089.451590] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.456504] pending: cpu_latency_work [ptlrpc] [ 2089.458957] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2089.461631] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2089.466345] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.468161] pending: kfree_rcu_monitor [ 2089.469215] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2089.472857] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2089.501022] workqueue events_long: flags=0x0 [ 2089.502421] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.513487] pending: lu_objects_flush [obdclass] [ 2089.515701] workqueue events_unbound: flags=0x2 [ 2089.525231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2089.527022] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2089.530321] workqueue events_power_efficient: flags=0x80 [ 2089.532086] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.535315] pending: neigh_periodic_work [ 2089.537369] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2089.539708] pending: do_cache_clean, neigh_periodic_work [ 2089.542804] workqueue mm_percpu_wq: flags=0x8 [ 2089.544178] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.555617] pending: lru_add_drain_per_cpu [ 2089.568147] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2089.576926] pending: vmstat_update, lru_add_drain_per_cpu [ 2089.584752] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2089.589688] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2089.593170] workqueue writeback: flags=0x4a [ 2089.595578] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2089.599265] pending: wb_workfn, wb_workfn [ 2089.602134] workqueue xprtiod: flags=0xa [ 2089.604030] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2089.609040] pending: xs_stream_data_receive_workfn [ 2089.614119] workqueue knbd0-recv: flags=0x1a [ 2089.616827] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2089.620259] in-flight: 655:recv_work [ 2089.622960] workqueue ptlrpc_pinger: flags=0x2 [ 2089.624548] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2089.630405] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2089.636783] pool 8: cpus=0-3 flags=0x4 nice=0 hung=64s workers=4 idle: 36396 38 11188 [ 2089.643834] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2080s workers=2 idle: 658 [ 2100.052160] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2100.054027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2100.063161] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2100.063305] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2100.063314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2100.063317] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2100.063335] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2100.063340] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2100.063344] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2100.063346] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2100.063348] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2100.063350] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2100.063352] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2100.063355] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2100.063357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2100.063359] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2100.063368] Call Trace: [ 2100.063387] [ 2100.063394] ? show_regs.cold.9+0x22/0x2f [ 2100.063405] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2100.063411] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2100.063416] ? watchdog+0x40/0x40 [ 2100.063421] ? hrtimer_interrupt+0x128/0x2c0 [ 2100.069401] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2100.069570] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2100.069577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2100.069579] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2100.069593] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2100.069597] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2100.069602] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2100.069604] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2100.069607] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2100.069609] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2100.069611] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2100.069615] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2100.069618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2100.069620] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2100.069628] Call Trace: [ 2100.069634] [ 2100.069638] ? show_regs.cold.9+0x22/0x2f [ 2100.069648] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2100.069654] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2100.069658] ? watchdog+0x40/0x40 [ 2100.069663] ? hrtimer_interrupt+0x128/0x2c0 [ 2100.069667] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2100.069678] ? apic_timer_interrupt+0xf/0x20 [ 2100.069740] [ 2100.069743] ? native_safe_halt+0x2e/0x50 [ 2100.069747] kvm_wait+0xcd/0xe0 [ 2100.069755] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2100.069763] do_raw_spin_lock+0x177/0x190 [ 2100.069767] _raw_spin_lock+0x21/0x30 [ 2100.069771] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2100.070099] ? do_raw_spin_unlock+0x75/0x190 [ 2100.070105] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2100.090312] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2100.090335] ? apic_timer_interrupt+0xf/0x20 [ 2100.090340] [ 2100.090344] ? native_safe_halt+0x2e/0x50 [ 2100.090348] kvm_wait+0xcd/0xe0 [ 2100.090357] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2100.090363] do_raw_spin_lock+0x177/0x190 [ 2100.090368] _raw_spin_lock+0x21/0x30 [ 2100.090375] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2100.090800] ? do_raw_spin_unlock+0x75/0x190 [ 2100.090807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2100.091013] ? do_wait_intr+0xf0/0xf0 [ 2100.091024] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2100.091379] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2100.091613] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2100.091913] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2100.098132] ? do_wait_intr+0xf0/0xf0 [ 2100.098154] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2100.104946] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2100.107985] mdc_close+0x53b/0xd70 [mdc] [ 2100.108034] lmv_close+0x18d/0x490 [lmv] [ 2100.108069] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2100.108286] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2100.121911] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2100.121975] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2100.122072] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2100.122263] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2100.126606] ll_md_close+0x274/0x800 [lustre] [ 2100.131430] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2100.131493] lmv_intent_open+0x338/0x1150 [lmv] [ 2100.131527] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2100.131637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2100.136150] ll_file_release+0x128/0xc50 [lustre] [ 2100.138973] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2100.139023] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2100.139145] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2100.142424] __fput+0x10d/0x440 [ 2100.142442] ____fput+0x12/0x20 [ 2100.142447] task_work_run+0xc6/0x110 [ 2100.142455] exit_to_usermode_loop+0x1dd/0x1f0 [ 2100.142462] do_syscall_64+0x3d6/0x440 [ 2100.142466] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2100.142474] RIP: 0033:0x7fdc0afbfcd8 [ 2100.142519] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2100.142525] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2100.142531] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2100.142533] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2100.142535] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2100.142538] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2100.142540] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2100.623291] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2100.625500] ? do_raw_spin_unlock+0x75/0x190 [ 2100.627814] ? _raw_spin_unlock+0x12/0x30 [ 2100.629902] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2100.636744] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2100.639059] ll_atomic_open+0x36a/0xd70 [lustre] [ 2100.652116] lookup_open+0x356/0xb40 [ 2100.655581] do_last+0x313/0xfc0 [ 2100.657966] ? nd_jump_root+0xe5/0x160 [ 2100.662906] ? path_init+0x437/0x520 [ 2100.667227] path_openat+0xf7/0x500 [ 2100.668760] ? do_raw_spin_unlock+0x75/0x190 [ 2100.670796] do_filp_open+0x99/0x140 [ 2100.675797] ? getname_flags+0x6e/0x330 [ 2100.677285] ? __check_object_size+0xff/0x256 [ 2100.678806] ? do_raw_spin_unlock+0x75/0x190 [ 2100.685391] ? _raw_spin_unlock+0x12/0x30 [ 2100.689759] do_sys_openat2+0x2b4/0x410 [ 2100.692843] do_sys_open+0x73/0xa0 [ 2100.694524] __x64_sys_openat+0x24/0x30 [ 2100.696830] do_syscall_64+0xc1/0x440 [ 2100.699591] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2100.706702] RIP: 0033:0x7fdc0afbf332 [ 2100.711791] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2100.725904] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2100.733479] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2100.735970] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2100.760799] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2100.767895] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2100.772924] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2104.053188] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2104.063158] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2104.124941] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2104.137387] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2104.145454] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2104.152433] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2104.156030] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2104.165920] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2104.169795] RAX: 0000000000007690 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2104.172840] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2104.175915] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2104.178869] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2104.181967] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2104.185081] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2104.188828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2104.191181] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2104.194417] Call Trace: [ 2104.195469] [ 2104.196377] ? show_regs.cold.9+0x22/0x2f [ 2104.197987] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2104.200000] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2104.201895] ? watchdog+0x40/0x40 [ 2104.203212] ? hrtimer_interrupt+0x128/0x2c0 [ 2104.205008] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2104.206943] ? apic_timer_interrupt+0xf/0x20 [ 2104.208807] [ 2104.209749] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2104.211964] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2104.214209] do_raw_spin_lock+0x177/0x190 [ 2104.215932] _raw_spin_lock+0x21/0x30 [ 2104.217554] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2104.220189] process_one_work+0x2c8/0x700 [ 2104.222422] worker_thread+0x296/0x6e0 [ 2104.224009] ? rescuer_thread+0x570/0x570 [ 2104.225685] kthread+0x1d1/0x200 [ 2104.227466] ? set_kthread_struct+0x70/0x70 [ 2104.228981] ret_from_fork+0x1f/0x30 [ 2116.055293] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2116.065021] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2116.114306] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2116.123488] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2116.129955] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2116.131529] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2116.157524] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2116.162122] RAX: 0000000000007293 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2116.166442] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2116.173199] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2116.177718] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2116.184004] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2116.186696] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2116.191584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2116.195535] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2116.197626] Call Trace: [ 2116.198386] [ 2116.205994] ? show_regs.cold.9+0x22/0x2f [ 2116.208699] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2116.215624] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2116.217889] ? watchdog+0x40/0x40 [ 2116.220673] ? hrtimer_interrupt+0x128/0x2c0 [ 2116.222692] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2116.228176] ? apic_timer_interrupt+0xf/0x20 [ 2116.231543] [ 2116.234798] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2116.239591] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2116.246174] ? dump_stack+0xf6/0x10e [ 2116.247574] do_raw_spin_lock+0x177/0x190 [ 2116.251924] _raw_spin_lock+0x21/0x30 [ 2116.254795] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2116.259347] obd_mod_free+0x46/0x140 [ptlrpc] [ 2116.264832] mdc_commit_open+0x91/0xb0 [mdc] [ 2116.269850] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2116.274678] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2116.280162] after_reply+0xe2c/0x1780 [ptlrpc] [ 2116.287263] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2116.294113] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2116.298771] ? do_wait_intr+0xf0/0xf0 [ 2116.301190] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2116.305844] kthread+0x1d1/0x200 [ 2116.309260] ? set_kthread_struct+0x70/0x70 [ 2116.312728] ret_from_fork+0x1f/0x30 [ 2120.159125] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 95s! [ 2120.162062] Showing busy workqueues and worker pools: [ 2120.165125] workqueue events: flags=0x0 [ 2120.166871] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2120.169634] pending: cpu_latency_work [ptlrpc] [ 2120.171978] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2120.174911] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2120.178607] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2120.181662] pending: kfree_rcu_monitor [ 2120.183663] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2120.186726] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2120.190572] workqueue events_long: flags=0x0 [ 2120.192665] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2120.195173] pending: lu_objects_flush [obdclass] [ 2120.197600] workqueue events_unbound: flags=0x2 [ 2120.199816] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2120.202581] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2120.206870] workqueue events_power_efficient: flags=0x80 [ 2120.209731] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2120.213021] pending: neigh_periodic_work, check_lifetime [ 2120.215153] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2120.217770] pending: do_cache_clean, neigh_periodic_work [ 2120.220526] workqueue mm_percpu_wq: flags=0x8 [ 2120.222680] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2120.225156] pending: lru_add_drain_per_cpu [ 2120.227178] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2120.230279] pending: vmstat_update, lru_add_drain_per_cpu [ 2120.232958] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2120.235760] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2120.238675] workqueue writeback: flags=0x4a [ 2120.240715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2120.243619] pending: wb_workfn, wb_workfn [ 2120.245289] workqueue xprtiod: flags=0xa [ 2120.247138] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2120.249838] pending: xs_stream_data_receive_workfn [ 2120.253150] workqueue ipv6_addrconf: flags=0x40008 [ 2120.255573] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2120.257904] pending: addrconf_verify_work [ 2120.259896] workqueue knbd0-recv: flags=0x1a [ 2120.261356] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2120.263942] in-flight: 655:recv_work [ 2120.265674] workqueue ptlrpc_pinger: flags=0x2 [ 2120.267599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2120.270514] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2120.273178] pool 8: cpus=0-3 flags=0x4 nice=0 hung=95s workers=4 idle: 36396 38 11188 [ 2120.276683] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2110s workers=2 idle: 658 [ 2128.052336] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2128.054077] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2128.056851] Modules linked in: [ 2128.065558] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2128.065986] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2128.065994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2128.065997] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2128.066011] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2128.066015] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2128.066020] RAX: 0000000000007f85 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2128.066023] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2128.066025] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2128.066027] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2128.066029] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2128.066033] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2128.066036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2128.066038] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2128.066047] Call Trace: [ 2128.066053] [ 2128.066059] ? show_regs.cold.9+0x22/0x2f [ 2128.066068] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2128.066075] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2128.066080] ? watchdog+0x40/0x40 [ 2128.066084] ? hrtimer_interrupt+0x128/0x2c0 [ 2128.066088] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2128.066099] ? apic_timer_interrupt+0xf/0x20 [ 2128.066104] [ 2128.066108] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2128.066114] do_raw_spin_lock+0x177/0x190 [ 2128.066119] _raw_spin_lock+0x21/0x30 [ 2128.066124] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2128.066544] ? do_raw_spin_unlock+0x75/0x190 [ 2128.066550] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2128.071526] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2128.071671] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2128.071677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2128.071680] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2128.071693] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2128.071698] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2128.071702] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2128.071704] RDX: 0000000000003a14 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2128.071707] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2128.071709] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2128.071711] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2128.071714] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2128.071717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2128.071719] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2128.071727] Call Trace: [ 2128.071732] [ 2128.071738] ? show_regs.cold.9+0x22/0x2f [ 2128.071747] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2128.071753] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2128.071757] ? watchdog+0x40/0x40 [ 2128.071761] ? hrtimer_interrupt+0x128/0x2c0 [ 2128.071765] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2128.071773] ? apic_timer_interrupt+0xf/0x20 [ 2128.071777] [ 2128.071780] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2128.071785] do_raw_spin_lock+0x177/0x190 [ 2128.071790] _raw_spin_lock+0x21/0x30 [ 2128.071796] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2128.072315] ? do_raw_spin_unlock+0x75/0x190 [ 2128.072322] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2128.106758] ? do_wait_intr+0xf0/0xf0 [ 2128.111988] ? do_wait_intr+0xf0/0xf0 [ 2128.112010] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2128.112291] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2128.112509] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2128.112733] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2128.112844] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2128.112882] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2128.112912] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2128.112947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2128.113034] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2128.113398] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2128.113437] lmv_intent_open+0x338/0x1150 [lmv] [ 2128.113466] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2128.113560] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2128.113655] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2128.113683] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2128.113867] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2128.113966] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2128.114050] ? do_raw_spin_unlock+0x75/0x190 [ 2128.114056] ? _raw_spin_unlock+0x12/0x30 [ 2128.114063] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2128.114090] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2128.114768] ll_atomic_open+0x36a/0xd70 [lustre] [ 2128.114862] lookup_open+0x356/0xb40 [ 2128.114870] do_last+0x313/0xfc0 [ 2128.114876] ? nd_jump_root+0xe5/0x160 [ 2128.114880] ? path_init+0x437/0x520 [ 2128.114892] path_openat+0xf7/0x500 [ 2128.114897] ? do_raw_spin_unlock+0x75/0x190 [ 2128.114902] do_filp_open+0x99/0x140 [ 2128.114912] ? getname_flags+0x6e/0x330 [ 2128.114917] ? __check_object_size+0xff/0x256 [ 2128.114927] ? do_raw_spin_unlock+0x75/0x190 [ 2128.114931] ? _raw_spin_unlock+0x12/0x30 [ 2128.114936] do_sys_openat2+0x2b4/0x410 [ 2128.114941] do_sys_open+0x73/0xa0 [ 2128.114945] __x64_sys_openat+0x24/0x30 [ 2128.114949] do_syscall_64+0xc1/0x440 [ 2128.114955] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2128.114961] RIP: 0033:0x7fdc0afbf332 [ 2128.114968] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2128.114972] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2128.114977] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2128.114979] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2128.114981] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2128.114983] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2128.114985] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2128.955355] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2128.967793] mdc_close+0x53b/0xd70 [mdc] [ 2128.978769] lmv_close+0x18d/0x490 [lmv] [ 2128.983812] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2128.996116] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2129.008383] ll_md_close+0x274/0x800 [lustre] [ 2129.011666] ll_file_release+0x128/0xc50 [lustre] [ 2129.018838] __fput+0x10d/0x440 [ 2129.028786] ____fput+0x12/0x20 [ 2129.034784] task_work_run+0xc6/0x110 [ 2129.041329] exit_to_usermode_loop+0x1dd/0x1f0 [ 2129.048644] do_syscall_64+0x3d6/0x440 [ 2129.052153] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2129.061682] RIP: 0033:0x7fdc0afbfcd8 [ 2129.068615] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2129.092684] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2129.103488] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2129.108889] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2129.119538] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2129.130531] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2129.145784] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2132.053205] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2132.059889] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2132.116040] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2132.131233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2132.142006] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2132.147261] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2132.151141] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2132.164754] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2132.169643] RAX: 000000000000070d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2132.173419] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2132.178729] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2132.183370] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2132.188309] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2132.193305] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2132.198458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2132.202009] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2132.206719] Call Trace: [ 2132.208103] [ 2132.209033] ? show_regs.cold.9+0x22/0x2f [ 2132.212454] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2132.216747] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2132.219404] ? watchdog+0x40/0x40 [ 2132.222076] ? hrtimer_interrupt+0x128/0x2c0 [ 2132.224678] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2132.229182] ? apic_timer_interrupt+0xf/0x20 [ 2132.231852] [ 2132.232993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2132.235925] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2132.239098] do_raw_spin_lock+0x177/0x190 [ 2132.244826] _raw_spin_lock+0x21/0x30 [ 2132.247458] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2132.255094] process_one_work+0x2c8/0x700 [ 2132.261454] worker_thread+0x296/0x6e0 [ 2132.265371] ? rescuer_thread+0x570/0x570 [ 2132.274967] kthread+0x1d1/0x200 [ 2132.279028] ? set_kthread_struct+0x70/0x70 [ 2132.283211] ret_from_fork+0x1f/0x30 [ 2148.050081] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2148.062462] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2148.131788] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2148.147947] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2148.158244] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2148.162805] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2148.174238] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2148.183168] RAX: 0000000000007c8b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2148.190218] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2148.194951] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2148.198947] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2148.211944] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2148.224533] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2148.234177] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.240441] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2148.248227] Call Trace: [ 2148.254166] [ 2148.254931] ? show_regs.cold.9+0x22/0x2f [ 2148.263291] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2148.269201] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2148.274210] ? watchdog+0x40/0x40 [ 2148.277276] ? hrtimer_interrupt+0x128/0x2c0 [ 2148.291254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2148.297946] ? apic_timer_interrupt+0xf/0x20 [ 2148.302284] [ 2148.303537] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2148.307270] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2148.310646] ? dump_stack+0xf6/0x10e [ 2148.315366] do_raw_spin_lock+0x177/0x190 [ 2148.319222] _raw_spin_lock+0x21/0x30 [ 2148.323702] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2148.327098] obd_mod_free+0x46/0x140 [ptlrpc] [ 2148.329668] mdc_commit_open+0x91/0xb0 [mdc] [ 2148.334093] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2148.343696] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2148.353460] after_reply+0xe2c/0x1780 [ptlrpc] [ 2148.358141] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2148.364701] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2148.369959] ? do_wait_intr+0xf0/0xf0 [ 2148.373111] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2148.380722] kthread+0x1d1/0x200 [ 2148.385080] ? set_kthread_struct+0x70/0x70 [ 2148.388335] ret_from_fork+0x1f/0x30 [ 2150.369507] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 125s! [ 2150.387098] Showing busy workqueues and worker pools: [ 2150.407064] workqueue events: flags=0x0 [ 2150.417233] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2150.433972] pending: cpu_latency_work [ptlrpc] [ 2150.442013] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2150.447947] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2150.466642] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2150.472444] pending: kfree_rcu_monitor [ 2150.476834] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2150.495446] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2150.510099] workqueue events_long: flags=0x0 [ 2150.515341] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2150.523725] pending: lu_objects_flush [obdclass] [ 2150.532146] workqueue events_unbound: flags=0x2 [ 2150.544259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2150.558685] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2150.561856] workqueue events_power_efficient: flags=0x80 [ 2150.573765] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2150.590246] pending: neigh_periodic_work, check_lifetime [ 2150.608848] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2150.624559] pending: do_cache_clean, neigh_periodic_work [ 2150.638866] workqueue mm_percpu_wq: flags=0x8 [ 2150.651897] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2150.669016] pending: lru_add_drain_per_cpu [ 2150.672839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2150.687645] pending: vmstat_update, lru_add_drain_per_cpu [ 2150.705269] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2150.715516] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2150.720502] workqueue writeback: flags=0x4a [ 2150.724608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2150.726995] pending: wb_workfn, wb_workfn [ 2150.736721] workqueue xprtiod: flags=0xa [ 2150.745270] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2150.750146] pending: xs_stream_data_receive_workfn [ 2150.763541] workqueue ipv6_addrconf: flags=0x40008 [ 2150.779993] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2150.786701] pending: addrconf_verify_work [ 2150.795866] workqueue knbd0-recv: flags=0x1a [ 2150.800348] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2150.802213] in-flight: 655:recv_work [ 2150.809653] workqueue ptlrpc_pinger: flags=0x2 [ 2150.813788] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2150.819392] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2150.822995] pool 8: cpus=0-3 flags=0x4 nice=0 hung=125s workers=4 idle: 36396 38 11188 [ 2150.827441] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2141s workers=2 idle: 658 [ 2156.052184] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2156.056520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2156.056874] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2156.098908] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2156.098921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2156.098923] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2156.098937] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2156.098941] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2156.098947] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2156.098949] RDX: 00000000000057b9 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2156.098952] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2156.098954] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2156.098957] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2156.098960] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2156.098963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2156.098965] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2156.098973] Call Trace: [ 2156.098978] [ 2156.098982] ? show_regs.cold.9+0x22/0x2f [ 2156.098990] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2156.098996] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2156.099000] ? watchdog+0x40/0x40 [ 2156.099005] ? hrtimer_interrupt+0x128/0x2c0 [ 2156.099009] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2156.099017] ? apic_timer_interrupt+0xf/0x20 [ 2156.099022] [ 2156.099025] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2156.099031] do_raw_spin_lock+0x177/0x190 [ 2156.099035] _raw_spin_lock+0x21/0x30 [ 2156.099042] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2156.099901] ? do_raw_spin_unlock+0x75/0x190 [ 2156.099909] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2156.100415] ? do_wait_intr+0xf0/0xf0 [ 2156.100425] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2156.100995] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2156.101627] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2156.101993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2156.102308] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2156.102346] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2156.102373] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2156.102399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2156.102490] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2156.103042] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2156.103075] lmv_intent_open+0x338/0x1150 [lmv] [ 2156.103290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2156.103398] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2156.103494] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2156.103590] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2156.103689] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2156.103907] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2156.104004] ? do_raw_spin_unlock+0x75/0x190 [ 2156.104373] ? _raw_spin_unlock+0x12/0x30 [ 2156.104380] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2156.104408] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2156.104431] ll_atomic_open+0x36a/0xd70 [lustre] [ 2156.104601] lookup_open+0x356/0xb40 [ 2156.104609] do_last+0x313/0xfc0 [ 2156.104614] ? nd_jump_root+0xe5/0x160 [ 2156.104618] ? path_init+0x437/0x520 [ 2156.104623] path_openat+0xf7/0x500 [ 2156.104628] ? do_raw_spin_unlock+0x75/0x190 [ 2156.104632] do_filp_open+0x99/0x140 [ 2156.104639] ? getname_flags+0x6e/0x330 [ 2156.104643] ? __check_object_size+0xff/0x256 [ 2156.104648] ? do_raw_spin_unlock+0x75/0x190 [ 2156.104652] ? _raw_spin_unlock+0x12/0x30 [ 2156.104656] do_sys_openat2+0x2b4/0x410 [ 2156.104660] do_sys_open+0x73/0xa0 [ 2156.104664] __x64_sys_openat+0x24/0x30 [ 2156.104668] do_syscall_64+0xc1/0x440 [ 2156.104674] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2156.104680] RIP: 0033:0x7fdc0afbf332 [ 2156.104687] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2156.104690] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2156.104695] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2156.104698] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2156.104918] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2156.104920] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2156.104922] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2156.599994] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2156.668301] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2156.683541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2156.691664] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2156.697978] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2156.727306] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2156.732461] RAX: 0000000000007f14 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2156.738249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2156.747716] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2156.753929] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2156.758258] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2156.765191] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2156.773661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2156.782564] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2156.791531] Call Trace: [ 2156.794342] [ 2156.795503] ? show_regs.cold.9+0x22/0x2f [ 2156.798930] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2156.803256] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2156.806780] ? watchdog+0x40/0x40 [ 2156.808680] ? hrtimer_interrupt+0x128/0x2c0 [ 2156.811033] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2156.814110] ? apic_timer_interrupt+0xf/0x20 [ 2156.816335] [ 2156.817560] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2156.821776] do_raw_spin_lock+0x177/0x190 [ 2156.825063] _raw_spin_lock+0x21/0x30 [ 2156.827047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2156.830599] ? do_raw_spin_unlock+0x75/0x190 [ 2156.833486] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2156.836858] ? do_wait_intr+0xf0/0xf0 [ 2156.840025] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2156.846554] mdc_close+0x53b/0xd70 [mdc] [ 2156.849508] lmv_close+0x18d/0x490 [lmv] [ 2156.859288] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2156.870029] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2156.888244] ll_md_close+0x274/0x800 [lustre] [ 2156.899727] ll_file_release+0x128/0xc50 [lustre] [ 2156.903013] __fput+0x10d/0x440 [ 2156.909654] ____fput+0x12/0x20 [ 2156.912554] task_work_run+0xc6/0x110 [ 2156.921431] exit_to_usermode_loop+0x1dd/0x1f0 [ 2156.925715] do_syscall_64+0x3d6/0x440 [ 2156.927027] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2156.936018] RIP: 0033:0x7fdc0afbfcd8 [ 2156.940545] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2156.962037] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2156.967589] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2156.971850] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2156.975921] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2156.979985] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2156.984571] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2160.052052] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2160.062688] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2160.097434] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2160.110193] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2160.117564] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2160.125018] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2160.128160] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2160.141427] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2160.145586] RAX: 0000000000007c45 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2160.149382] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2160.153067] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2160.157311] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2160.166355] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2160.173479] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2160.179370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2160.183186] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2160.190753] Call Trace: [ 2160.194858] [ 2160.195582] ? show_regs.cold.9+0x22/0x2f [ 2160.197344] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2160.202710] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2160.208185] ? watchdog+0x40/0x40 [ 2160.209546] ? hrtimer_interrupt+0x128/0x2c0 [ 2160.215359] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2160.218146] ? apic_timer_interrupt+0xf/0x20 [ 2160.222510] [ 2160.223424] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2160.229248] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2160.231260] do_raw_spin_lock+0x177/0x190 [ 2160.236692] _raw_spin_lock+0x21/0x30 [ 2160.238815] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2160.244855] process_one_work+0x2c8/0x700 [ 2160.250286] worker_thread+0x296/0x6e0 [ 2160.252438] ? rescuer_thread+0x570/0x570 [ 2160.257685] kthread+0x1d1/0x200 [ 2160.259400] ? set_kthread_struct+0x70/0x70 [ 2160.264510] ret_from_fork+0x1f/0x30 [ 2176.052038] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2176.065625] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2176.123321] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2176.138437] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2176.154780] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.159584] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2176.174995] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2176.181898] RAX: 0000000000007d04 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2176.187488] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2176.193866] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2176.197027] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2176.199520] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2176.202942] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2176.206996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2176.210770] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2176.217457] Call Trace: [ 2176.219036] [ 2176.220616] ? show_regs.cold.9+0x22/0x2f [ 2176.222553] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2176.227515] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2176.230916] ? watchdog+0x40/0x40 [ 2176.233851] ? hrtimer_interrupt+0x128/0x2c0 [ 2176.237772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2176.244568] ? apic_timer_interrupt+0xf/0x20 [ 2176.249818] [ 2176.251021] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.256029] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2176.258096] ? dump_stack+0xf6/0x10e [ 2176.263875] do_raw_spin_lock+0x177/0x190 [ 2176.266035] _raw_spin_lock+0x21/0x30 [ 2176.270692] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2176.275963] obd_mod_free+0x46/0x140 [ptlrpc] [ 2176.278963] mdc_commit_open+0x91/0xb0 [mdc] [ 2176.283015] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2176.290062] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2176.300108] after_reply+0xe2c/0x1780 [ptlrpc] [ 2176.302400] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2176.312086] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2176.319732] ? do_wait_intr+0xf0/0xf0 [ 2176.320932] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2176.328974] kthread+0x1d1/0x200 [ 2176.331152] ? set_kthread_struct+0x70/0x70 [ 2176.334907] ret_from_fork+0x1f/0x30 [ 2181.087887] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 156s! [ 2181.093157] Showing busy workqueues and worker pools: [ 2181.104614] workqueue events: flags=0x0 [ 2181.107800] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2181.114729] pending: cpu_latency_work [ptlrpc] [ 2181.118726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2181.124011] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2181.130736] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2181.136903] pending: kfree_rcu_monitor [ 2181.141665] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2181.147788] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2181.158734] workqueue events_long: flags=0x0 [ 2181.160335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2181.169580] pending: lu_objects_flush [obdclass] [ 2181.174794] workqueue events_unbound: flags=0x2 [ 2181.176904] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2181.182420] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2181.188276] workqueue events_power_efficient: flags=0x80 [ 2181.193652] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2181.196809] pending: neigh_periodic_work, check_lifetime [ 2181.199599] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2181.204382] pending: do_cache_clean, neigh_periodic_work [ 2181.206624] workqueue mm_percpu_wq: flags=0x8 [ 2181.208121] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2181.211398] pending: lru_add_drain_per_cpu [ 2181.213111] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2181.215998] pending: vmstat_update, lru_add_drain_per_cpu [ 2181.218464] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2181.220879] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2181.223321] workqueue writeback: flags=0x4a [ 2181.224861] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2181.227134] pending: wb_workfn, wb_workfn [ 2181.228830] workqueue xprtiod: flags=0xa [ 2181.230341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2181.232605] pending: xs_stream_data_receive_workfn [ 2181.234810] workqueue ipv6_addrconf: flags=0x40008 [ 2181.236581] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2181.239034] pending: addrconf_verify_work [ 2181.240659] workqueue knbd0-recv: flags=0x1a [ 2181.242183] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2181.244123] in-flight: 655:recv_work [ 2181.245325] workqueue ptlrpc_pinger: flags=0x2 [ 2181.246755] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2181.248979] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2181.251322] pool 8: cpus=0-3 flags=0x4 nice=0 hung=156s workers=4 idle: 36396 38 11188 [ 2181.254095] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2171s workers=2 idle: 658 [ 2184.052011] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 2184.054755] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 2184.055857] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2184.055948] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2184.055960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2184.055964] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2184.055983] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2184.055986] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2184.055990] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2184.055992] RDX: 000000000000533b RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2184.055998] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2184.056000] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2184.056003] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2184.056005] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2184.056008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2184.056010] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2184.056120] Call Trace: [ 2184.056126] [ 2184.056132] ? show_regs.cold.9+0x22/0x2f [ 2184.059981] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 2184.097055] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2184.097074] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2184.097078] ? watchdog+0x40/0x40 [ 2184.097082] ? hrtimer_interrupt+0x128/0x2c0 [ 2184.097086] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2184.097094] ? apic_timer_interrupt+0xf/0x20 [ 2184.097098] [ 2184.097103] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2184.097116] do_raw_spin_lock+0x177/0x190 [ 2184.097127] _raw_spin_lock+0x21/0x30 [ 2184.097139] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2184.097492] ? do_raw_spin_unlock+0x75/0x190 [ 2184.097498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2184.097764] ? do_wait_intr+0xf0/0xf0 [ 2184.097807] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2184.098092] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2184.098330] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2184.098577] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2184.098719] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2184.098757] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2184.098787] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2184.098815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2184.098994] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2184.099282] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2184.099312] lmv_intent_open+0x338/0x1150 [lmv] [ 2184.099355] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2184.099443] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2184.099532] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2184.099556] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2184.099659] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2184.099746] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2184.099836] ? do_raw_spin_unlock+0x75/0x190 [ 2184.099843] ? _raw_spin_unlock+0x12/0x30 [ 2184.099850] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2184.099875] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2184.099897] ll_atomic_open+0x36a/0xd70 [lustre] [ 2184.099986] lookup_open+0x356/0xb40 [ 2184.099995] do_last+0x313/0xfc0 [ 2184.100002] ? nd_jump_root+0xe5/0x160 [ 2184.100008] ? path_init+0x437/0x520 [ 2184.100012] path_openat+0xf7/0x500 [ 2184.100017] ? do_raw_spin_unlock+0x75/0x190 [ 2184.100021] do_filp_open+0x99/0x140 [ 2184.100027] ? getname_flags+0x6e/0x330 [ 2184.100031] ? __check_object_size+0xff/0x256 [ 2184.100038] ? do_raw_spin_unlock+0x75/0x190 [ 2184.100042] ? _raw_spin_unlock+0x12/0x30 [ 2184.100047] do_sys_openat2+0x2b4/0x410 [ 2184.100050] do_sys_open+0x73/0xa0 [ 2184.100054] __x64_sys_openat+0x24/0x30 [ 2184.100057] do_syscall_64+0xc1/0x440 [ 2184.100063] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2184.100069] RIP: 0033:0x7fdc0afbf332 [ 2184.100076] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2184.100079] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2184.100085] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2184.100088] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2184.100090] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2184.100093] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2184.100095] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2184.481862] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2184.516977] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2184.522568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2184.528695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2184.534982] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2184.559477] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2184.564131] RAX: 0000000000007c85 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2184.569345] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2184.575027] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2184.578294] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2184.589704] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2184.596517] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2184.604107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2184.608463] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2184.611846] Call Trace: [ 2184.612954] [ 2184.613855] ? show_regs.cold.9+0x22/0x2f [ 2184.616104] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2184.627677] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2184.630892] ? watchdog+0x40/0x40 [ 2184.632560] ? hrtimer_interrupt+0x128/0x2c0 [ 2184.642070] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2184.644579] ? apic_timer_interrupt+0xf/0x20 [ 2184.648702] [ 2184.650900] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2184.653949] do_raw_spin_lock+0x177/0x190 [ 2184.659020] _raw_spin_lock+0x21/0x30 [ 2184.662718] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2184.665834] ? do_raw_spin_unlock+0x75/0x190 [ 2184.670572] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2184.675956] ? do_wait_intr+0xf0/0xf0 [ 2184.678488] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2184.683900] mdc_close+0x53b/0xd70 [mdc] [ 2184.686599] lmv_close+0x18d/0x490 [lmv] [ 2184.693054] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2184.696988] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2184.701837] ll_md_close+0x274/0x800 [lustre] [ 2184.704923] ll_file_release+0x128/0xc50 [lustre] [ 2184.709542] __fput+0x10d/0x440 [ 2184.711454] ____fput+0x12/0x20 [ 2184.713030] task_work_run+0xc6/0x110 [ 2184.718770] exit_to_usermode_loop+0x1dd/0x1f0 [ 2184.725572] do_syscall_64+0x3d6/0x440 [ 2184.728476] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2184.730244] RIP: 0033:0x7fdc0afbfcd8 [ 2184.736970] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2184.748624] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2184.758156] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2184.765580] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2184.770769] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2184.776022] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2184.779912] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2188.051062] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2188.054012] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2188.075238] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2188.079668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2188.083248] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2188.086214] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2188.088147] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2188.096123] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2188.099812] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2188.102491] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2188.104490] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2188.109219] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2188.112440] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2188.115641] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2188.119261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2188.122221] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2188.125307] Call Trace: [ 2188.126254] [ 2188.126954] ? show_regs.cold.9+0x22/0x2f [ 2188.128510] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2188.130361] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2188.132320] ? watchdog+0x40/0x40 [ 2188.133598] ? hrtimer_interrupt+0x128/0x2c0 [ 2188.135455] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2188.137697] ? apic_timer_interrupt+0xf/0x20 [ 2188.139459] [ 2188.140433] ? native_safe_halt+0x2e/0x50 [ 2188.142875] kvm_wait+0xcd/0xe0 [ 2188.144127] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2188.145969] do_raw_spin_lock+0x177/0x190 [ 2188.147842] _raw_spin_lock+0x21/0x30 [ 2188.149605] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2188.152080] process_one_work+0x2c8/0x700 [ 2188.153991] worker_thread+0x296/0x6e0 [ 2188.156104] ? rescuer_thread+0x570/0x570 [ 2188.157430] kthread+0x1d1/0x200 [ 2188.158504] ? set_kthread_struct+0x70/0x70 [ 2188.159818] ret_from_fork+0x1f/0x30 [ 2208.048865] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2208.053480] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2208.094559] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2208.100139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2208.109873] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2208.117148] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2208.139575] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2208.148304] RAX: 0000000000006b82 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2208.153510] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2208.158347] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2208.172551] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2208.180340] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2208.195947] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2208.201392] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2208.204497] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2208.212097] Call Trace: [ 2208.212963] [ 2208.220507] ? show_regs.cold.9+0x22/0x2f [ 2208.228485] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2208.234902] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2208.242171] ? watchdog+0x40/0x40 [ 2208.252901] ? hrtimer_interrupt+0x128/0x2c0 [ 2208.260789] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2208.262540] ? apic_timer_interrupt+0xf/0x20 [ 2208.267871] [ 2208.271044] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2208.283236] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2208.285432] ? dump_stack+0xf6/0x10e [ 2208.287561] do_raw_spin_lock+0x177/0x190 [ 2208.289431] _raw_spin_lock+0x21/0x30 [ 2208.291526] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2208.294554] obd_mod_free+0x46/0x140 [ptlrpc] [ 2208.297466] mdc_commit_open+0x91/0xb0 [mdc] [ 2208.300281] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2208.309112] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2208.314744] after_reply+0xe2c/0x1780 [ptlrpc] [ 2208.319071] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2208.332678] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2208.334128] ? do_wait_intr+0xf0/0xf0 [ 2208.342163] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2208.348808] kthread+0x1d1/0x200 [ 2208.350445] ? set_kthread_struct+0x70/0x70 [ 2208.353285] ret_from_fork+0x1f/0x30 [ 2211.295073] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 186s! [ 2211.297750] Showing busy workqueues and worker pools: [ 2211.300130] workqueue events: flags=0x0 [ 2211.301869] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2211.305129] pending: cpu_latency_work [ptlrpc] [ 2211.307938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2211.311048] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2211.315262] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2211.318264] pending: kfree_rcu_monitor [ 2211.320371] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2211.323721] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2211.328191] workqueue events_long: flags=0x0 [ 2211.330351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2211.333303] pending: lu_objects_flush [obdclass] [ 2211.336092] workqueue events_unbound: flags=0x2 [ 2211.338751] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2211.341367] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2211.344297] workqueue events_power_efficient: flags=0x80 [ 2211.346686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2211.349172] pending: neigh_periodic_work, check_lifetime [ 2211.351911] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2211.354937] pending: do_cache_clean, neigh_periodic_work [ 2211.357580] workqueue mm_percpu_wq: flags=0x8 [ 2211.359537] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2211.362838] pending: lru_add_drain_per_cpu [ 2211.364860] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2211.368589] pending: vmstat_update, lru_add_drain_per_cpu [ 2211.371543] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2211.374810] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2211.377899] workqueue writeback: flags=0x4a [ 2211.380039] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2211.383552] pending: wb_workfn, wb_workfn [ 2211.385556] workqueue xprtiod: flags=0xa [ 2211.387799] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2211.390993] pending: xs_stream_data_receive_workfn [ 2211.393265] workqueue ipv6_addrconf: flags=0x40008 [ 2211.395985] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2211.399844] pending: addrconf_verify_work [ 2211.402228] workqueue knbd0-recv: flags=0x1a [ 2211.404994] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2211.408927] in-flight: 655:recv_work [ 2211.410909] workqueue ptlrpc_pinger: flags=0x2 [ 2211.413086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2211.416181] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2211.419470] pool 8: cpus=0-3 flags=0x4 nice=0 hung=186s workers=4 idle: 36396 38 11188 [ 2211.422427] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2201s workers=2 idle: 658 [ 2212.052074] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 2212.055576] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 2212.061094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2212.061193] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2212.061200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2212.061203] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2212.061230] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2212.061239] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2212.061244] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2212.061245] RDX: 000000000000384c RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2212.061248] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2212.061255] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2212.061257] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2212.061260] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2212.061262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2212.061265] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2212.061273] Call Trace: [ 2212.061278] [ 2212.061284] ? show_regs.cold.9+0x22/0x2f [ 2212.061305] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2212.061312] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2212.061317] ? watchdog+0x40/0x40 [ 2212.061321] ? hrtimer_interrupt+0x128/0x2c0 [ 2212.061325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2212.061333] ? apic_timer_interrupt+0xf/0x20 [ 2212.061338] [ 2212.061342] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2212.061347] do_raw_spin_lock+0x177/0x190 [ 2212.061352] _raw_spin_lock+0x21/0x30 [ 2212.061358] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2212.061665] ? do_raw_spin_unlock+0x75/0x190 [ 2212.061671] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2212.062107] ? do_wait_intr+0xf0/0xf0 [ 2212.062117] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2212.062471] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2212.062705] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2212.062994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2212.063112] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2212.063172] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2212.063199] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2212.063364] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2212.078653] Modules linked in: lustre(O) [ 2212.104017] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2212.106031] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2212.108120] lmv_intent_open+0x338/0x1150 [lmv] [ 2212.108239] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2212.125498] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2212.125579] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2212.125586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2212.132306] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2212.132436] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2212.132476] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2212.132583] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2212.132693] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2212.132875] ? do_raw_spin_unlock+0x75/0x190 [ 2212.132883] ? _raw_spin_unlock+0x12/0x30 [ 2212.132895] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2212.132919] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2212.132946] ll_atomic_open+0x36a/0xd70 [lustre] [ 2212.133255] lookup_open+0x356/0xb40 [ 2212.133270] do_last+0x313/0xfc0 [ 2212.133276] ? nd_jump_root+0xe5/0x160 [ 2212.133281] ? path_init+0x437/0x520 [ 2212.133285] path_openat+0xf7/0x500 [ 2212.133290] ? do_raw_spin_unlock+0x75/0x190 [ 2212.133380] do_filp_open+0x99/0x140 [ 2212.133385] ? getname_flags+0x6e/0x330 [ 2212.133390] ? __check_object_size+0xff/0x256 [ 2212.133396] ? do_raw_spin_unlock+0x75/0x190 [ 2212.133401] ? _raw_spin_unlock+0x12/0x30 [ 2212.133404] do_sys_openat2+0x2b4/0x410 [ 2212.133408] do_sys_open+0x73/0xa0 [ 2212.133412] __x64_sys_openat+0x24/0x30 [ 2212.133416] do_syscall_64+0xc1/0x440 [ 2212.133422] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2212.133428] RIP: 0033:0x7fdc0afbf332 [ 2212.133436] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2212.133441] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2212.133445] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2212.133447] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2212.133449] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2212.133451] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2212.133453] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2212.699973] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2212.701905] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2212.738556] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2212.750171] RAX: 0000000000001619 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2212.762052] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2212.767424] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2212.787708] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2212.799135] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2212.808665] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2212.829101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2212.841000] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2212.852801] Call Trace: [ 2212.854635] [ 2212.858332] ? show_regs.cold.9+0x22/0x2f [ 2212.867280] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2212.868771] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2212.877382] ? watchdog+0x40/0x40 [ 2212.883296] ? hrtimer_interrupt+0x128/0x2c0 [ 2212.890944] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2212.897160] ? apic_timer_interrupt+0xf/0x20 [ 2212.898652] [ 2212.899393] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2212.915370] do_raw_spin_lock+0x177/0x190 [ 2212.916801] _raw_spin_lock+0x21/0x30 [ 2212.926593] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2212.934166] ? do_raw_spin_unlock+0x75/0x190 [ 2212.938638] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2212.949594] ? do_wait_intr+0xf0/0xf0 [ 2212.950861] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2212.954311] mdc_close+0x53b/0xd70 [mdc] [ 2212.955544] lmv_close+0x18d/0x490 [lmv] [ 2212.963579] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2212.977464] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2212.979060] ll_md_close+0x274/0x800 [lustre] [ 2212.980532] ll_file_release+0x128/0xc50 [lustre] [ 2212.998287] __fput+0x10d/0x440 [ 2212.999379] ____fput+0x12/0x20 [ 2213.009647] task_work_run+0xc6/0x110 [ 2213.020401] exit_to_usermode_loop+0x1dd/0x1f0 [ 2213.028996] do_syscall_64+0x3d6/0x440 [ 2213.030871] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2213.039865] RIP: 0033:0x7fdc0afbfcd8 [ 2213.044251] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2213.074191] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2213.091269] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2213.093588] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2213.108288] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2213.122364] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2213.134387] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2216.051054] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2216.056287] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2216.090522] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2216.097117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2216.102892] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2216.106547] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2216.109901] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2216.120268] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2216.124621] RAX: 00000000000072ce RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2216.128118] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2216.131817] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2216.135630] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2216.139664] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2216.145069] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2216.148656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2216.151486] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2216.154977] Call Trace: [ 2216.156235] [ 2216.157224] ? show_regs.cold.9+0x22/0x2f [ 2216.159087] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2216.161308] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2216.163579] ? watchdog+0x40/0x40 [ 2216.165464] ? hrtimer_interrupt+0x128/0x2c0 [ 2216.167558] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2216.169893] ? apic_timer_interrupt+0xf/0x20 [ 2216.172044] [ 2216.173131] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2216.175572] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2216.178152] do_raw_spin_lock+0x177/0x190 [ 2216.180100] _raw_spin_lock+0x21/0x30 [ 2216.182103] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2216.184885] process_one_work+0x2c8/0x700 [ 2216.186819] worker_thread+0x296/0x6e0 [ 2216.188756] ? rescuer_thread+0x570/0x570 [ 2216.190731] kthread+0x1d1/0x200 [ 2216.192344] ? set_kthread_struct+0x70/0x70 [ 2216.194295] ret_from_fork+0x1f/0x30 [ 2236.051331] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2236.058121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2236.097364] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2236.114977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2236.123006] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2236.126049] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2236.140227] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2236.145287] RAX: 0000000000007e37 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2236.151358] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2236.156002] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2236.162235] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2236.166025] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2236.172432] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2236.176627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2236.180324] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2236.184413] Call Trace: [ 2236.186030] [ 2236.187203] ? show_regs.cold.9+0x22/0x2f [ 2236.191613] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2236.197075] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2236.201035] ? watchdog+0x40/0x40 [ 2236.204063] ? hrtimer_interrupt+0x128/0x2c0 [ 2236.210232] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2236.211423] ? apic_timer_interrupt+0xf/0x20 [ 2236.216450] [ 2236.217484] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2236.224456] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2236.230952] ? dump_stack+0xf6/0x10e [ 2236.236556] do_raw_spin_lock+0x177/0x190 [ 2236.238081] _raw_spin_lock+0x21/0x30 [ 2236.251941] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2236.259216] obd_mod_free+0x46/0x140 [ptlrpc] [ 2236.264148] mdc_commit_open+0x91/0xb0 [mdc] [ 2236.268175] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2236.275605] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2236.279984] after_reply+0xe2c/0x1780 [ptlrpc] [ 2236.285170] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2236.291926] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2236.296509] ? do_wait_intr+0xf0/0xf0 [ 2236.299350] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2236.303110] kthread+0x1d1/0x200 [ 2236.307423] ? set_kthread_struct+0x70/0x70 [ 2236.311197] ret_from_fork+0x1f/0x30 [ 2240.052064] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 2240.054031] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 2240.055946] Modules linked in: lustre(O) [ 2240.059488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2240.059629] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2240.059646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2240.059649] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2240.059663] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2240.059667] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2240.059672] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2240.059679] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2240.059681] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2240.059683] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2240.059685] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2240.059688] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2240.059691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2240.059693] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2240.059701] Call Trace: [ 2240.059706] [ 2240.059712] ? show_regs.cold.9+0x22/0x2f [ 2240.059722] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2240.059728] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2240.059732] ? watchdog+0x40/0x40 [ 2240.059736] ? hrtimer_interrupt+0x128/0x2c0 [ 2240.059739] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2240.059745] ? apic_timer_interrupt+0xf/0x20 [ 2240.059749] [ 2240.059751] ? native_safe_halt+0x2e/0x50 [ 2240.059756] kvm_wait+0xcd/0xe0 [ 2240.059762] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2240.059771] do_raw_spin_lock+0x177/0x190 [ 2240.059775] _raw_spin_lock+0x21/0x30 [ 2240.059778] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2240.060407] ? do_raw_spin_unlock+0x75/0x190 [ 2240.060413] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2240.062898] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2240.062976] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2240.063086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2240.063089] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2240.063101] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2240.063105] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2240.063109] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2240.063111] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2240.063113] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2240.063115] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2240.063117] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2240.063119] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2240.063127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2240.063129] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2240.063137] Call Trace: [ 2240.063142] [ 2240.063148] ? show_regs.cold.9+0x22/0x2f [ 2240.088129] ? do_wait_intr+0xf0/0xf0 [ 2240.088217] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2240.088688] mdc_close+0x53b/0xd70 [mdc] [ 2240.088725] lmv_close+0x18d/0x490 [lmv] [ 2240.088764] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2240.089005] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2240.089087] ll_md_close+0x274/0x800 [lustre] [ 2240.089446] ll_file_release+0x128/0xc50 [lustre] [ 2240.089547] __fput+0x10d/0x440 [ 2240.089564] ____fput+0x12/0x20 [ 2240.089569] task_work_run+0xc6/0x110 [ 2240.089578] exit_to_usermode_loop+0x1dd/0x1f0 [ 2240.089585] do_syscall_64+0x3d6/0x440 [ 2240.089590] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2240.089603] RIP: 0033:0x7fdc0afbfcd8 [ 2240.089610] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2240.089614] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2240.089619] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2240.089621] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2240.089624] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2240.089626] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2240.089631] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2240.352516] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2240.356685] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2240.359530] ? watchdog+0x40/0x40 [ 2240.361442] ? hrtimer_interrupt+0x128/0x2c0 [ 2240.363707] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2240.366384] ? apic_timer_interrupt+0xf/0x20 [ 2240.368958] [ 2240.370367] ? native_safe_halt+0x2e/0x50 [ 2240.372950] kvm_wait+0xcd/0xe0 [ 2240.374551] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2240.377478] do_raw_spin_lock+0x177/0x190 [ 2240.379550] _raw_spin_lock+0x21/0x30 [ 2240.381621] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2240.384831] ? do_raw_spin_unlock+0x75/0x190 [ 2240.386957] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2240.390236] ? do_wait_intr+0xf0/0xf0 [ 2240.391983] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2240.394764] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2240.397885] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2240.400921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2240.402875] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2240.405041] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2240.406370] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2240.408123] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2240.410354] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2240.413382] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2240.415471] lmv_intent_open+0x338/0x1150 [lmv] [ 2240.417989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2240.421101] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2240.423951] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2240.426417] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2240.428421] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2240.432556] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2240.435713] ? do_raw_spin_unlock+0x75/0x190 [ 2240.439017] ? _raw_spin_unlock+0x12/0x30 [ 2240.441434] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2240.444643] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2240.450636] ll_atomic_open+0x36a/0xd70 [lustre] [ 2240.453394] lookup_open+0x356/0xb40 [ 2240.454950] do_last+0x313/0xfc0 [ 2240.456825] ? nd_jump_root+0xe5/0x160 [ 2240.458761] ? path_init+0x437/0x520 [ 2240.460926] path_openat+0xf7/0x500 [ 2240.462637] ? do_raw_spin_unlock+0x75/0x190 [ 2240.465389] do_filp_open+0x99/0x140 [ 2240.467063] ? getname_flags+0x6e/0x330 [ 2240.468989] ? __check_object_size+0xff/0x256 [ 2240.471823] ? do_raw_spin_unlock+0x75/0x190 [ 2240.474355] ? _raw_spin_unlock+0x12/0x30 [ 2240.477404] do_sys_openat2+0x2b4/0x410 [ 2240.479316] do_sys_open+0x73/0xa0 [ 2240.481549] __x64_sys_openat+0x24/0x30 [ 2240.483448] do_syscall_64+0xc1/0x440 [ 2240.485606] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2240.488436] RIP: 0033:0x7fdc0afbf332 [ 2240.490449] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2240.500983] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2240.504488] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2240.510604] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2240.513547] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2240.516653] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2240.521409] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2241.506077] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 216s! [ 2241.513054] Showing busy workqueues and worker pools: [ 2241.519117] workqueue events: flags=0x0 [ 2241.522912] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2241.530186] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2241.537159] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2241.542145] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2241.546646] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2241.558890] pending: kfree_rcu_monitor [ 2241.562437] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2241.570335] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2241.578252] workqueue events_long: flags=0x0 [ 2241.582319] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2241.584522] pending: lu_objects_flush [obdclass] [ 2241.589028] workqueue events_unbound: flags=0x2 [ 2241.592118] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2241.595871] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2241.600916] workqueue events_power_efficient: flags=0x80 [ 2241.604995] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2241.611173] pending: neigh_periodic_work, check_lifetime [ 2241.615083] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2241.618747] pending: do_cache_clean, neigh_periodic_work [ 2241.625554] workqueue mm_percpu_wq: flags=0x8 [ 2241.629148] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2241.632833] pending: lru_add_drain_per_cpu [ 2241.638532] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2241.646027] pending: vmstat_update, lru_add_drain_per_cpu [ 2241.648893] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2241.652453] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2241.658749] workqueue writeback: flags=0x4a [ 2241.660659] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2241.664552] pending: wb_workfn, wb_workfn [ 2241.666708] workqueue xprtiod: flags=0xa [ 2241.668967] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2241.672867] pending: xs_stream_data_receive_workfn [ 2241.675442] workqueue ipv6_addrconf: flags=0x40008 [ 2241.680712] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2241.684388] pending: addrconf_verify_work [ 2241.686931] workqueue knbd0-recv: flags=0x1a [ 2241.689405] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2241.694429] in-flight: 655:recv_work [ 2241.697047] workqueue ptlrpc_pinger: flags=0x2 [ 2241.702652] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2241.711888] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2241.715575] pool 8: cpus=0-3 flags=0x4 nice=0 hung=216s workers=4 idle: 36396 38 11188 [ 2241.724660] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2232s workers=2 idle: 658 [ 2244.052106] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2244.057286] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2244.095860] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2244.105246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2244.112010] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2244.116721] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2244.121238] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2244.135695] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2244.146516] RAX: 0000000000007a19 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2244.152054] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2244.157772] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2244.167430] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2244.173796] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2244.183761] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2244.197177] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2244.202200] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2244.210084] Call Trace: [ 2244.211907] [ 2244.217106] ? show_regs.cold.9+0x22/0x2f [ 2244.220099] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2244.230054] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2244.234823] ? watchdog+0x40/0x40 [ 2244.238240] ? hrtimer_interrupt+0x128/0x2c0 [ 2244.240110] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2244.246345] ? apic_timer_interrupt+0xf/0x20 [ 2244.248648] [ 2244.253038] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2244.257966] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2244.262973] do_raw_spin_lock+0x177/0x190 [ 2244.266754] _raw_spin_lock+0x21/0x30 [ 2244.269625] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2244.277536] process_one_work+0x2c8/0x700 [ 2244.279662] worker_thread+0x296/0x6e0 [ 2244.281462] ? rescuer_thread+0x570/0x570 [ 2244.284213] kthread+0x1d1/0x200 [ 2244.286572] ? set_kthread_struct+0x70/0x70 [ 2244.290009] ret_from_fork+0x1f/0x30 [ 2259.351028] rcu: INFO: rcu_sched self-detected stall on CPU [ 2259.353992] rcu: 0-...!: (144963 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 2259.360645] (t=240008 jiffies g=270761 q=2692) [ 2259.363004] rcu: rcu_sched kthread starved for 234940 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 2259.369242] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2259.374447] rcu: RCU grace-period kthread stack dump: [ 2259.376831] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2259.382238] Call Trace: [ 2259.383718] __schedule+0x351/0xcb0 [ 2259.385743] schedule+0xc0/0x180 [ 2259.387358] schedule_timeout+0xb4/0x190 [ 2259.389185] ? do_raw_spin_unlock+0x75/0x190 [ 2259.391650] ? __next_timer_interrupt+0x160/0x160 [ 2259.394286] ? prepare_to_swait_event+0xf3/0x140 [ 2259.396857] rcu_gp_kthread+0x8dd/0x15b0 [ 2259.398933] ? force_qs_rnp+0x2c0/0x2c0 [ 2259.401239] kthread+0x1d1/0x200 [ 2259.402642] ? set_kthread_struct+0x70/0x70 [ 2259.404137] ret_from_fork+0x1f/0x30 [ 2259.406163] rcu: Stack dump where RCU GP kthread last ran: [ 2259.409586] Sending NMI from CPU 0 to CPUs 3: [ 2259.415285] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 2.537 msecs [ 2259.416008] NMI backtrace for cpu 3 [ 2259.416012] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2259.416014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2259.416016] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2259.416017] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2259.416020] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 [ 2259.416022] RAX: 0000000000007b52 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2259.416024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2259.416025] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2259.416027] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2259.416028] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2259.416029] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2259.416030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2259.416031] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2259.416032] Call Trace: [ 2259.416033] [ 2259.416034] ? show_regs.cold.9+0x22/0x2f [ 2259.416035] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2259.416036] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2259.416037] ? nmi_handle+0xbd/0x210 [ 2259.416038] ? default_do_nmi+0x4b/0x1b0 [ 2259.416039] ? do_nmi+0x280/0x380 [ 2259.416040] ? end_repeat_nmi+0x16/0x1f [ 2259.416041] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2259.416042] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2259.416043] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2259.416044] [ 2259.416045] do_raw_spin_lock+0x177/0x190 [ 2259.416046] _raw_spin_lock+0x21/0x30 [ 2259.416047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2259.416049] ? do_raw_spin_unlock+0x75/0x190 [ 2259.416050] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2259.416051] ? do_wait_intr+0xf0/0xf0 [ 2259.416052] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2259.416053] mdc_close+0x53b/0xd70 [mdc] [ 2259.416054] lmv_close+0x18d/0x490 [lmv] [ 2259.416055] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2259.416056] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2259.416057] ll_md_close+0x274/0x800 [lustre] [ 2259.416059] ll_file_release+0x128/0xc50 [lustre] [ 2259.416060] __fput+0x10d/0x440 [ 2259.416061] ____fput+0x12/0x20 [ 2259.416061] task_work_run+0xc6/0x110 [ 2259.416062] exit_to_usermode_loop+0x1dd/0x1f0 [ 2259.416063] do_syscall_64+0x3d6/0x440 [ 2259.416064] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2259.416066] RIP: 0033:0x7fdc0afbfcd8 [ 2259.416067] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2259.416069] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2259.416072] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2259.416073] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2259.416075] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2259.416076] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2259.416078] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2259.416083] NMI backtrace for cpu 0 [ 2259.681267] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2259.696277] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2259.706726] Call Trace: [ 2259.708747] [ 2259.715199] ? dump_stack+0xbb/0x10e [ 2259.719111] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2259.731202] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2259.733041] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2259.738085] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2259.743108] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2259.756701] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2259.767122] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2259.777556] ? update_process_times+0x6d/0xc0 [ 2259.785474] ? tick_sched_handle.isra.18+0x27/0x90 [ 2259.789131] ? tick_sched_timer+0x53/0xb0 [ 2259.800209] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2259.802604] ? tick_sched_handle.isra.18+0x90/0x90 [ 2259.804871] ? hrtimer_interrupt+0x128/0x2c0 [ 2259.806894] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2259.809897] ? apic_timer_interrupt+0xf/0x20 [ 2259.812236] [ 2259.813399] ? native_safe_halt+0x2e/0x50 [ 2259.815480] ? kvm_wait+0xcd/0xe0 [ 2259.816994] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2259.819703] ? dump_stack+0xf6/0x10e [ 2259.821588] ? do_raw_spin_lock+0x177/0x190 [ 2259.823856] ? _raw_spin_lock+0x21/0x30 [ 2259.825788] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2259.829416] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2259.833834] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2259.838254] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2259.843113] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2259.847671] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2259.850573] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2259.854156] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2259.859120] ? do_wait_intr+0xf0/0xf0 [ 2259.861534] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2259.865423] ? kthread+0x1d1/0x200 [ 2259.867849] ? set_kthread_struct+0x70/0x70 [ 2259.871772] ? ret_from_fork+0x1f/0x30 [ 2268.053101] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 2268.065499] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2268.143073] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2268.160617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2268.176066] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2268.189906] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2268.216803] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2268.223895] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2268.243547] RDX: 00000000000014f1 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2268.250960] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2268.264698] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2268.271439] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2268.281008] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2268.290803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2268.297941] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2268.310619] Call Trace: [ 2268.317983] [ 2268.324038] ? show_regs.cold.9+0x22/0x2f [ 2268.325497] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2268.342885] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2268.344200] ? watchdog+0x40/0x40 [ 2268.351359] ? hrtimer_interrupt+0x128/0x2c0 [ 2268.357445] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2268.364712] ? apic_timer_interrupt+0xf/0x20 [ 2268.368909] [ 2268.371257] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2268.375693] do_raw_spin_lock+0x177/0x190 [ 2268.378633] _raw_spin_lock+0x21/0x30 [ 2268.381233] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2268.388848] ? do_raw_spin_unlock+0x75/0x190 [ 2268.391835] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2268.397850] ? do_wait_intr+0xf0/0xf0 [ 2268.400804] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2268.410385] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2268.419523] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2268.427901] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2268.431833] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2268.437535] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2268.440397] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2268.446632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2268.450709] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2268.456745] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2268.463043] lmv_intent_open+0x338/0x1150 [lmv] [ 2268.466428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2268.471625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2268.476031] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2268.482207] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2268.489866] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2268.497701] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2268.503548] ? do_raw_spin_unlock+0x75/0x190 [ 2268.507929] ? _raw_spin_unlock+0x12/0x30 [ 2268.514006] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2268.517990] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2268.523216] ll_atomic_open+0x36a/0xd70 [lustre] [ 2268.527054] lookup_open+0x356/0xb40 [ 2268.531889] do_last+0x313/0xfc0 [ 2268.536399] ? nd_jump_root+0xe5/0x160 [ 2268.540720] ? path_init+0x437/0x520 [ 2268.545373] path_openat+0xf7/0x500 [ 2268.546675] ? do_raw_spin_unlock+0x75/0x190 [ 2268.549865] do_filp_open+0x99/0x140 [ 2268.552240] ? getname_flags+0x6e/0x330 [ 2268.554871] ? __check_object_size+0xff/0x256 [ 2268.565167] ? do_raw_spin_unlock+0x75/0x190 [ 2268.568409] ? _raw_spin_unlock+0x12/0x30 [ 2268.573630] do_sys_openat2+0x2b4/0x410 [ 2268.579152] do_sys_open+0x73/0xa0 [ 2268.584206] __x64_sys_openat+0x24/0x30 [ 2268.589387] do_syscall_64+0xc1/0x440 [ 2268.595286] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2268.598191] RIP: 0033:0x7fdc0afbf332 [ 2268.601721] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2268.620360] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2268.628230] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2268.637821] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2268.645995] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2268.649235] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2268.658658] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2272.056687] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 2272.081816] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2272.176374] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2272.203023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2272.214583] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2272.225173] BUG: workqueue lockup - pool [ 2272.231158] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2272.231182] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2272.231187] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 [ 2272.233189] cpus=0-3 flags=0x4 nice=0 stuck for 247s! [ 2272.233208] Showing busy workqueues and worker pools: [ 2272.233211] workqueue events: flags=0x0 [ 2272.233214] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2272.233223] pending: cpu_latency_work [ptlrpc] [ 2272.248929] ORIG_RAX: ffffffffffffff13 [ 2272.248939] RAX: 000000000000480a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2272.248942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2272.248944] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2272.248946] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2272.248948] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2272.248951] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2272.248953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2272.248956] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2272.248965] Call Trace: [ 2272.248971] [ 2272.248976] ? show_regs.cold.9+0x22/0x2f [ 2272.248989] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2272.248995] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2272.248999] ? watchdog+0x40/0x40 [ 2272.249003] ? hrtimer_interrupt+0x128/0x2c0 [ 2272.249006] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2272.249014] ? apic_timer_interrupt+0xf/0x20 [ 2272.249019] [ 2272.249021] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2272.249028] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2272.249036] do_raw_spin_lock+0x177/0x190 [ 2272.249041] _raw_spin_lock+0x21/0x30 [ 2272.249045] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2272.249324] process_one_work+0x2c8/0x700 [ 2272.249330] worker_thread+0x296/0x6e0 [ 2272.249334] ? rescuer_thread+0x570/0x570 [ 2272.249338] kthread+0x1d1/0x200 [ 2272.249342] ? set_kthread_struct+0x70/0x70 [ 2272.249346] ret_from_fork+0x1f/0x30 [ 2272.474293] , push_to_pool [ 2272.475733] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2272.483394] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2272.491572] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2272.495622] pending: kfree_rcu_monitor [ 2272.498291] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2272.502234] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2272.507440] workqueue events_long: flags=0x0 [ 2272.509684] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2272.513129] pending: lu_objects_flush [obdclass] [ 2272.515932] workqueue events_unbound: flags=0x2 [ 2272.518255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2272.521425] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2272.526574] workqueue events_power_efficient: flags=0x80 [ 2272.535205] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2272.542715] pending: neigh_periodic_work, check_lifetime [ 2272.547205] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2272.550565] pending: do_cache_clean, neigh_periodic_work [ 2272.558879] workqueue mm_percpu_wq: flags=0x8 [ 2272.563468] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2272.579806] pending: lru_add_drain_per_cpu [ 2272.588884] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2272.600433] pending: vmstat_update, lru_add_drain_per_cpu [ 2272.608712] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2272.618735] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2272.628518] workqueue writeback: flags=0x4a [ 2272.634611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2272.643382] pending: wb_workfn, wb_workfn [ 2272.648291] workqueue xprtiod: flags=0xa [ 2272.656275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2272.663268] pending: xs_stream_data_receive_workfn [ 2272.671293] workqueue ipv6_addrconf: flags=0x40008 [ 2272.680600] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2272.687633] pending: addrconf_verify_work [ 2272.707253] workqueue knbd0-recv: flags=0x1a [ 2272.715649] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2272.725304] in-flight: 655:recv_work [ 2272.730333] workqueue ptlrpc_pinger: flags=0x2 [ 2272.732521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2272.739891] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2272.746430] pool 8: cpus=0-3 flags=0x4 nice=0 hung=247s workers=4 idle: 36396 38 11188 [ 2272.756214] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2263s workers=2 idle: 658 [ 2284.054125] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 2284.057195] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2284.092750] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2284.113076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2284.118312] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2284.124007] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2284.136763] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2284.145113] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2284.152137] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2284.158679] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2284.165352] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2284.168562] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2284.177280] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2284.186015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2284.190527] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2284.193568] Call Trace: [ 2284.196128] [ 2284.198395] ? show_regs.cold.9+0x22/0x2f [ 2284.199714] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2284.203077] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2284.206001] ? watchdog+0x40/0x40 [ 2284.208315] ? hrtimer_interrupt+0x128/0x2c0 [ 2284.210250] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2284.212410] ? apic_timer_interrupt+0xf/0x20 [ 2284.213926] [ 2284.215985] ? native_safe_halt+0x2e/0x50 [ 2284.218429] kvm_wait+0xcd/0xe0 [ 2284.221115] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2284.225356] do_raw_spin_lock+0x177/0x190 [ 2284.227181] _raw_spin_lock+0x21/0x30 [ 2284.231838] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2284.236784] ? do_raw_spin_unlock+0x75/0x190 [ 2284.239237] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2284.241202] ? do_wait_intr+0xf0/0xf0 [ 2284.243393] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2284.248743] mdc_close+0x53b/0xd70 [mdc] [ 2284.250224] lmv_close+0x18d/0x490 [lmv] [ 2284.252288] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2284.255869] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2284.259464] ll_md_close+0x274/0x800 [lustre] [ 2284.261126] ll_file_release+0x128/0xc50 [lustre] [ 2284.265857] __fput+0x10d/0x440 [ 2284.266668] ____fput+0x12/0x20 [ 2284.267994] task_work_run+0xc6/0x110 [ 2284.270559] exit_to_usermode_loop+0x1dd/0x1f0 [ 2284.273849] do_syscall_64+0x3d6/0x440 [ 2284.276673] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2284.279729] RIP: 0033:0x7fdc0afbfcd8 [ 2284.285853] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2284.304620] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2284.312724] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2284.317159] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2284.324877] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2284.330569] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2284.342206] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2296.056158] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2296.064373] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2296.108128] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2296.114129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2296.124876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2296.130723] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2296.141318] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2296.145189] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2296.149974] RDX: 0000000000006f39 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2296.152479] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2296.160359] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2296.164030] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2296.169492] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2296.180171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2296.187585] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2296.196137] Call Trace: [ 2296.197162] [ 2296.198236] ? show_regs.cold.9+0x22/0x2f [ 2296.200157] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2296.202374] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2296.204132] ? watchdog+0x40/0x40 [ 2296.205571] ? hrtimer_interrupt+0x128/0x2c0 [ 2296.207094] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2296.208849] ? apic_timer_interrupt+0xf/0x20 [ 2296.210794] [ 2296.211712] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2296.213679] do_raw_spin_lock+0x177/0x190 [ 2296.215552] _raw_spin_lock+0x21/0x30 [ 2296.217134] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2296.219407] ? do_raw_spin_unlock+0x75/0x190 [ 2296.220980] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2296.223297] ? do_wait_intr+0xf0/0xf0 [ 2296.224761] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2296.226729] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2296.228825] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2296.231342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2296.237409] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2296.242618] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2296.247291] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2296.249570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2296.255104] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2296.257309] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2296.262950] lmv_intent_open+0x338/0x1150 [lmv] [ 2296.264501] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2296.279422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2296.283881] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2296.285454] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2296.292721] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2296.297091] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2296.299753] ? do_raw_spin_unlock+0x75/0x190 [ 2296.303387] ? _raw_spin_unlock+0x12/0x30 [ 2296.306458] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2296.316036] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2296.320385] ll_atomic_open+0x36a/0xd70 [lustre] [ 2296.326371] lookup_open+0x356/0xb40 [ 2296.327629] do_last+0x313/0xfc0 [ 2296.329804] ? nd_jump_root+0xe5/0x160 [ 2296.337508] ? path_init+0x437/0x520 [ 2296.339224] path_openat+0xf7/0x500 [ 2296.344178] ? do_raw_spin_unlock+0x75/0x190 [ 2296.345804] do_filp_open+0x99/0x140 [ 2296.354814] ? getname_flags+0x6e/0x330 [ 2296.357040] ? __check_object_size+0xff/0x256 [ 2296.363255] ? do_raw_spin_unlock+0x75/0x190 [ 2296.364894] ? _raw_spin_unlock+0x12/0x30 [ 2296.366214] do_sys_openat2+0x2b4/0x410 [ 2296.374444] do_sys_open+0x73/0xa0 [ 2296.375584] __x64_sys_openat+0x24/0x30 [ 2296.377912] do_syscall_64+0xc1/0x440 [ 2296.379425] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2296.387573] RIP: 0033:0x7fdc0afbf332 [ 2296.388884] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2296.411412] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2296.417302] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2296.425796] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2296.431457] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2296.441696] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2296.455385] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2300.048037] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 2300.050856] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2300.051024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 2300.082037] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2300.090516] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 2300.100759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2300.100776] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2300.100799] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2300.100804] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2300.100811] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2300.100814] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2300.100817] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2300.100821] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2300.118731] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2300.118782] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2300.118790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2300.118797] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2300.119274] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2300.119287] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2300.119292] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2300.119297] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2300.119299] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2300.119303] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2300.119305] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2300.119308] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2300.119312] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2300.119316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2300.119319] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2300.119328] Call Trace: [ 2300.119334] [ 2300.119340] ? show_regs.cold.9+0x22/0x2f [ 2300.119349] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2300.119356] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2300.119360] ? watchdog+0x40/0x40 [ 2300.119365] ? hrtimer_interrupt+0x128/0x2c0 [ 2300.119369] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2300.119376] ? apic_timer_interrupt+0xf/0x20 [ 2300.119381] [ 2300.119384] ? native_safe_halt+0x2e/0x50 [ 2300.119388] kvm_wait+0xcd/0xe0 [ 2300.119396] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2300.119404] do_raw_spin_lock+0x177/0x190 [ 2300.119488] _raw_spin_lock+0x21/0x30 [ 2300.119496] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2300.119785] process_one_work+0x2c8/0x700 [ 2300.119792] worker_thread+0x296/0x6e0 [ 2300.119797] ? rescuer_thread+0x570/0x570 [ 2300.119800] kthread+0x1d1/0x200 [ 2300.119805] ? set_kthread_struct+0x70/0x70 [ 2300.119808] ret_from_fork+0x1f/0x30 [ 2300.349877] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2300.355321] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2300.363873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2300.367033] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2300.369783] Call Trace: [ 2300.374897] [ 2300.376613] ? show_regs.cold.9+0x22/0x2f [ 2300.379083] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2300.381225] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2300.383094] ? watchdog+0x40/0x40 [ 2300.385184] ? hrtimer_interrupt+0x128/0x2c0 [ 2300.387096] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2300.390096] ? apic_timer_interrupt+0xf/0x20 [ 2300.392739] [ 2300.393558] ? native_safe_halt+0x2e/0x50 [ 2300.395896] kvm_wait+0xcd/0xe0 [ 2300.397733] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2300.400051] ? dump_stack+0xf6/0x10e [ 2300.401520] do_raw_spin_lock+0x177/0x190 [ 2300.403763] _raw_spin_lock+0x21/0x30 [ 2300.405406] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2300.408905] obd_mod_free+0x46/0x140 [ptlrpc] [ 2300.411751] mdc_commit_open+0x91/0xb0 [mdc] [ 2300.417846] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2300.423935] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2300.429051] after_reply+0xe2c/0x1780 [ptlrpc] [ 2300.431517] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2300.434079] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2300.436442] ? do_wait_intr+0xf0/0xf0 [ 2300.438162] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2300.440874] kthread+0x1d1/0x200 [ 2300.442430] ? set_kthread_struct+0x70/0x70 [ 2300.444784] ret_from_fork+0x1f/0x30 [ 2302.946015] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 278s! [ 2302.952851] Showing busy workqueues and worker pools: [ 2302.955640] workqueue events: flags=0x0 [ 2302.960816] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2302.964030] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2302.971837] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2302.974466] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2302.983673] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2302.988795] pending: kfree_rcu_monitor [ 2302.991919] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2302.998159] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2303.004864] workqueue events_long: flags=0x0 [ 2303.007558] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2303.011020] pending: lu_objects_flush [obdclass] [ 2303.015174] workqueue events_unbound: flags=0x2 [ 2303.018285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2303.021970] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2303.029081] workqueue events_power_efficient: flags=0x80 [ 2303.032243] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2303.036295] pending: neigh_periodic_work, check_lifetime [ 2303.040189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2303.044958] pending: do_cache_clean, neigh_periodic_work [ 2303.048292] workqueue mm_percpu_wq: flags=0x8 [ 2303.050645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2303.054223] pending: lru_add_drain_per_cpu [ 2303.056845] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2303.060489] pending: vmstat_update, lru_add_drain_per_cpu [ 2303.063257] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2303.066780] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2303.070391] workqueue writeback: flags=0x4a [ 2303.072241] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2303.075378] pending: wb_workfn, wb_workfn [ 2303.077751] workqueue xprtiod: flags=0xa [ 2303.079949] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2303.082947] pending: xs_stream_data_receive_workfn [ 2303.085664] workqueue ipv6_addrconf: flags=0x40008 [ 2303.088371] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2303.092285] pending: addrconf_verify_work [ 2303.095381] workqueue knbd0-recv: flags=0x1a [ 2303.098942] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2303.102944] in-flight: 655:recv_work [ 2303.106084] workqueue ptlrpc_pinger: flags=0x2 [ 2303.109667] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2303.112965] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2303.116373] pool 8: cpus=0-3 flags=0x4 nice=0 hung=278s workers=4 idle: 36396 38 11188 [ 2303.120898] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2293s workers=2 idle: 658 [ 2312.054029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 2312.058244] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2312.120353] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2312.132305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2312.148980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2312.164625] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2312.184633] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2312.198009] RAX: 0000000000003cc5 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2312.205124] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2312.212867] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2312.217437] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2312.220339] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2312.223684] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2312.227750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2312.230401] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2312.233020] Call Trace: [ 2312.233980] [ 2312.234807] ? show_regs.cold.9+0x22/0x2f [ 2312.237086] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2312.239810] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2312.242285] ? watchdog+0x40/0x40 [ 2312.243808] ? hrtimer_interrupt+0x128/0x2c0 [ 2312.246115] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2312.248807] ? apic_timer_interrupt+0xf/0x20 [ 2312.250797] [ 2312.251728] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2312.254028] do_raw_spin_lock+0x177/0x190 [ 2312.255841] _raw_spin_lock+0x21/0x30 [ 2312.257935] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2312.261057] ? do_raw_spin_unlock+0x75/0x190 [ 2312.262500] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2312.265448] ? do_wait_intr+0xf0/0xf0 [ 2312.266832] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2312.270666] mdc_close+0x53b/0xd70 [mdc] [ 2312.272340] lmv_close+0x18d/0x490 [lmv] [ 2312.274097] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2312.276627] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2312.279085] ll_md_close+0x274/0x800 [lustre] [ 2312.282193] ll_file_release+0x128/0xc50 [lustre] [ 2312.285218] __fput+0x10d/0x440 [ 2312.287502] ____fput+0x12/0x20 [ 2312.289530] task_work_run+0xc6/0x110 [ 2312.291121] exit_to_usermode_loop+0x1dd/0x1f0 [ 2312.293797] do_syscall_64+0x3d6/0x440 [ 2312.295920] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2312.298359] RIP: 0033:0x7fdc0afbfcd8 [ 2312.301051] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2312.313529] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2312.316704] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2312.319641] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2312.322523] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2312.325750] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2312.328447] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2324.055880] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2324.059544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2324.084677] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2324.090575] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2324.094643] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2324.098274] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2324.107559] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2324.111622] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2324.114389] RDX: 0000000000001cda RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2324.117489] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2324.121250] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2324.124895] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2324.128351] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2324.133337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2324.135160] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2324.138031] Call Trace: [ 2324.139356] [ 2324.140135] ? show_regs.cold.9+0x22/0x2f [ 2324.142401] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2324.144480] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2324.146360] ? watchdog+0x40/0x40 [ 2324.148568] ? hrtimer_interrupt+0x128/0x2c0 [ 2324.151116] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2324.153920] ? apic_timer_interrupt+0xf/0x20 [ 2324.155534] [ 2324.156946] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2324.159677] do_raw_spin_lock+0x177/0x190 [ 2324.161271] _raw_spin_lock+0x21/0x30 [ 2324.162886] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2324.165607] ? do_raw_spin_unlock+0x75/0x190 [ 2324.167530] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2324.170064] ? do_wait_intr+0xf0/0xf0 [ 2324.171526] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2324.175012] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2324.178378] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2324.181124] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2324.183254] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2324.187289] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2324.189311] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2324.191515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2324.194907] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2324.198681] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2324.203073] lmv_intent_open+0x338/0x1150 [lmv] [ 2324.205543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2324.208186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2324.211922] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2324.214579] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2324.217857] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2324.219840] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2324.222102] ? do_raw_spin_unlock+0x75/0x190 [ 2324.224179] ? _raw_spin_unlock+0x12/0x30 [ 2324.226063] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2324.228618] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2324.231068] ll_atomic_open+0x36a/0xd70 [lustre] [ 2324.233691] lookup_open+0x356/0xb40 [ 2324.236040] do_last+0x313/0xfc0 [ 2324.237916] ? nd_jump_root+0xe5/0x160 [ 2324.240034] ? path_init+0x437/0x520 [ 2324.241875] path_openat+0xf7/0x500 [ 2324.244609] ? do_raw_spin_unlock+0x75/0x190 [ 2324.246438] do_filp_open+0x99/0x140 [ 2324.248568] ? getname_flags+0x6e/0x330 [ 2324.250293] ? __check_object_size+0xff/0x256 [ 2324.252410] ? do_raw_spin_unlock+0x75/0x190 [ 2324.254302] ? _raw_spin_unlock+0x12/0x30 [ 2324.256318] do_sys_openat2+0x2b4/0x410 [ 2324.258078] do_sys_open+0x73/0xa0 [ 2324.259998] __x64_sys_openat+0x24/0x30 [ 2324.261963] do_syscall_64+0xc1/0x440 [ 2324.263688] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2324.266098] RIP: 0033:0x7fdc0afbf332 [ 2324.267735] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2324.277012] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2324.280489] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2324.283553] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2324.287240] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2324.290590] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2324.294685] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2328.048022] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 2328.051683] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2328.051966] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 2328.071175] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2328.071193] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2328.071196] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2328.071270] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2328.071275] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2328.071285] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2328.071287] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2328.071289] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2328.071291] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2328.071294] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2328.071297] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2328.071299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2328.071301] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2328.071309] Call Trace: [ 2328.071316] [ 2328.071322] ? show_regs.cold.9+0x22/0x2f [ 2328.071332] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2328.071339] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2328.071343] ? watchdog+0x40/0x40 [ 2328.071347] ? hrtimer_interrupt+0x128/0x2c0 [ 2328.071352] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2328.071358] ? apic_timer_interrupt+0xf/0x20 [ 2328.071363] [ 2328.071366] ? native_safe_halt+0x2e/0x50 [ 2328.071369] kvm_wait+0xcd/0xe0 [ 2328.071377] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2328.071383] ? dump_stack+0xf6/0x10e [ 2328.071388] do_raw_spin_lock+0x177/0x190 [ 2328.071392] _raw_spin_lock+0x21/0x30 [ 2328.071407] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2328.080019] Modules linked in: lustre(O) [ 2328.089798] obd_mod_free+0x46/0x140 [ptlrpc] [ 2328.095812] osp(O) [ 2328.101380] mdc_commit_open+0x91/0xb0 [mdc] [ 2328.101432] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2328.114917] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2328.114990] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2328.117185] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2328.117468] after_reply+0xe2c/0x1780 [ptlrpc] [ 2328.117665] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2328.118126] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2328.118333] ? do_wait_intr+0xf0/0xf0 [ 2328.118343] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2328.118545] kthread+0x1d1/0x200 [ 2328.118552] ? set_kthread_struct+0x70/0x70 [ 2328.118556] ret_from_fork+0x1f/0x30 [ 2328.368539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2328.378315] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2328.385869] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2328.391418] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2328.411102] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2328.419556] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2328.426536] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2328.432629] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2328.439388] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2328.448258] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2328.454786] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2328.459213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2328.464394] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2328.470548] Call Trace: [ 2328.475777] [ 2328.477777] ? show_regs.cold.9+0x22/0x2f [ 2328.481894] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2328.488270] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2328.494922] ? watchdog+0x40/0x40 [ 2328.501005] ? hrtimer_interrupt+0x128/0x2c0 [ 2328.506397] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2328.509131] ? apic_timer_interrupt+0xf/0x20 [ 2328.511965] [ 2328.513371] ? native_safe_halt+0x2e/0x50 [ 2328.517123] kvm_wait+0xcd/0xe0 [ 2328.518892] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2328.522028] do_raw_spin_lock+0x177/0x190 [ 2328.525322] _raw_spin_lock+0x21/0x30 [ 2328.527203] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2328.530222] process_one_work+0x2c8/0x700 [ 2328.534177] worker_thread+0x296/0x6e0 [ 2328.539992] ? rescuer_thread+0x570/0x570 [ 2328.548850] kthread+0x1d1/0x200 [ 2328.552234] ? set_kthread_struct+0x70/0x70 [ 2328.555271] ret_from_fork+0x1f/0x30 [ 2333.157460] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 308s! [ 2333.174930] Showing busy workqueues and worker pools: [ 2333.184493] workqueue events: flags=0x0 [ 2333.205753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2333.232879] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2333.235418] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2333.255430] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2333.274270] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2333.283242] pending: kfree_rcu_monitor [ 2333.295099] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2333.306199] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2333.327664] workqueue events_long: flags=0x0 [ 2333.328930] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2333.337255] pending: lu_objects_flush [obdclass] [ 2333.346036] workqueue events_unbound: flags=0x2 [ 2333.347762] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2333.357462] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2333.376950] workqueue events_power_efficient: flags=0x80 [ 2333.387257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2333.393923] pending: neigh_periodic_work, check_lifetime [ 2333.406452] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2333.412582] pending: do_cache_clean, neigh_periodic_work [ 2333.417264] workqueue mm_percpu_wq: flags=0x8 [ 2333.427964] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2333.436437] pending: lru_add_drain_per_cpu [ 2333.440764] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2333.451718] pending: vmstat_update, lru_add_drain_per_cpu [ 2333.461138] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2333.471896] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2333.479227] workqueue writeback: flags=0x4a [ 2333.488250] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2333.499754] pending: wb_workfn, wb_workfn [ 2333.505384] workqueue xprtiod: flags=0xa [ 2333.513167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2333.522017] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2333.532557] workqueue ipv6_addrconf: flags=0x40008 [ 2333.534323] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2333.540025] pending: addrconf_verify_work [ 2333.542148] workqueue knbd0-recv: flags=0x1a [ 2333.544270] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2333.546971] in-flight: 655:recv_work [ 2333.549158] workqueue ptlrpc_pinger: flags=0x2 [ 2333.551511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2333.555264] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2333.559229] pool 8: cpus=0-3 flags=0x4 nice=0 hung=308s workers=4 idle: 36396 38 11188 [ 2333.563748] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2324s workers=2 idle: 658 [ 2340.054011] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 2340.057699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2340.085658] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2340.091398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2340.096374] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2340.099463] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2340.109056] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2340.113321] RAX: 000000000000730f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2340.116982] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2340.120705] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2340.124241] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2340.127980] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2340.131160] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2340.135285] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2340.139033] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2340.142959] Call Trace: [ 2340.144449] [ 2340.145389] ? show_regs.cold.9+0x22/0x2f [ 2340.147648] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2340.149888] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2340.151730] ? watchdog+0x40/0x40 [ 2340.154914] ? hrtimer_interrupt+0x128/0x2c0 [ 2340.158793] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2340.164447] ? apic_timer_interrupt+0xf/0x20 [ 2340.168915] [ 2340.171827] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2340.174382] do_raw_spin_lock+0x177/0x190 [ 2340.176429] _raw_spin_lock+0x21/0x30 [ 2340.178057] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2340.181310] ? do_raw_spin_unlock+0x75/0x190 [ 2340.183461] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2340.186440] ? do_wait_intr+0xf0/0xf0 [ 2340.188067] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2340.190875] mdc_close+0x53b/0xd70 [mdc] [ 2340.192954] lmv_close+0x18d/0x490 [lmv] [ 2340.195147] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2340.198557] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2340.200986] ll_md_close+0x274/0x800 [lustre] [ 2340.203484] ll_file_release+0x128/0xc50 [lustre] [ 2340.207575] __fput+0x10d/0x440 [ 2340.209110] ____fput+0x12/0x20 [ 2340.210467] task_work_run+0xc6/0x110 [ 2340.212280] exit_to_usermode_loop+0x1dd/0x1f0 [ 2340.214501] do_syscall_64+0x3d6/0x440 [ 2340.216899] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2340.219076] RIP: 0033:0x7fdc0afbfcd8 [ 2340.220959] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2340.229347] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2340.232876] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2340.236008] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2340.239334] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2340.242800] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2340.246205] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2352.054211] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2352.062177] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2352.089248] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2352.094912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2352.104971] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2352.111490] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2352.126658] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2352.132991] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2352.135438] RDX: 0000000000004190 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2352.142519] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2352.147922] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2352.157427] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2352.163353] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2352.166716] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2352.170525] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2352.175599] Call Trace: [ 2352.176654] [ 2352.179237] ? show_regs.cold.9+0x22/0x2f [ 2352.180976] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2352.183592] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2352.185594] ? watchdog+0x40/0x40 [ 2352.189448] ? hrtimer_interrupt+0x128/0x2c0 [ 2352.192475] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2352.199742] ? apic_timer_interrupt+0xf/0x20 [ 2352.204844] [ 2352.208335] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2352.210968] do_raw_spin_lock+0x177/0x190 [ 2352.217865] _raw_spin_lock+0x21/0x30 [ 2352.220367] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2352.227451] ? do_raw_spin_unlock+0x75/0x190 [ 2352.234329] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2352.241972] ? do_wait_intr+0xf0/0xf0 [ 2352.251178] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2352.257611] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2352.263465] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2352.271012] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2352.283800] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2352.292368] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2352.299173] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2352.302696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2352.310711] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2352.317282] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2352.322353] lmv_intent_open+0x338/0x1150 [lmv] [ 2352.324974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2352.329529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2352.332267] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2352.334399] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2352.340441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2352.350845] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2352.352989] ? do_raw_spin_unlock+0x75/0x190 [ 2352.360480] ? _raw_spin_unlock+0x12/0x30 [ 2352.365576] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2352.375721] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2352.381281] ll_atomic_open+0x36a/0xd70 [lustre] [ 2352.387186] lookup_open+0x356/0xb40 [ 2352.389447] do_last+0x313/0xfc0 [ 2352.398032] ? nd_jump_root+0xe5/0x160 [ 2352.399351] ? path_init+0x437/0x520 [ 2352.401502] path_openat+0xf7/0x500 [ 2352.407648] ? do_raw_spin_unlock+0x75/0x190 [ 2352.411480] do_filp_open+0x99/0x140 [ 2352.415874] ? getname_flags+0x6e/0x330 [ 2352.420578] ? __check_object_size+0xff/0x256 [ 2352.430110] ? do_raw_spin_unlock+0x75/0x190 [ 2352.432616] ? _raw_spin_unlock+0x12/0x30 [ 2352.434559] do_sys_openat2+0x2b4/0x410 [ 2352.436460] do_sys_open+0x73/0xa0 [ 2352.438728] __x64_sys_openat+0x24/0x30 [ 2352.440978] do_syscall_64+0xc1/0x440 [ 2352.444335] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2352.449043] RIP: 0033:0x7fdc0afbf332 [ 2352.450962] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2352.465213] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2352.473901] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2352.484889] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2352.495854] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2352.504115] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2352.506480] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2356.051025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 2356.059738] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2356.112327] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2356.124008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2356.134396] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2356.138735] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2356.143049] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2356.166803] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2356.173065] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2356.179301] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2356.183940] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2356.186346] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2356.196097] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2356.202139] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2356.206459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2356.209372] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2356.212302] Call Trace: [ 2356.213148] [ 2356.214247] ? show_regs.cold.9+0x22/0x2f [ 2356.216636] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2356.219031] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2356.221614] ? watchdog+0x40/0x40 [ 2356.223125] ? hrtimer_interrupt+0x128/0x2c0 [ 2356.224812] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2356.229030] ? apic_timer_interrupt+0xf/0x20 [ 2356.231297] [ 2356.232416] ? native_safe_halt+0x2e/0x50 [ 2356.234301] kvm_wait+0xcd/0xe0 [ 2356.235614] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2356.237726] do_raw_spin_lock+0x177/0x190 [ 2356.239466] _raw_spin_lock+0x21/0x30 [ 2356.240986] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2356.243439] process_one_work+0x2c8/0x700 [ 2356.245093] worker_thread+0x296/0x6e0 [ 2356.246608] ? rescuer_thread+0x570/0x570 [ 2356.248352] kthread+0x1d1/0x200 [ 2356.249586] ? set_kthread_struct+0x70/0x70 [ 2356.251340] ret_from_fork+0x1f/0x30 [ 2360.048647] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2360.056963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2360.095820] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2360.105636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2360.108892] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2360.113551] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2360.132254] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2360.136939] RAX: 0000000000004984 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2360.146701] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2360.150428] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2360.154006] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2360.157940] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2360.161701] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2360.164994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2360.169211] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2360.173907] Call Trace: [ 2360.175933] [ 2360.177053] ? show_regs.cold.9+0x22/0x2f [ 2360.179678] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2360.185452] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2360.189686] ? watchdog+0x40/0x40 [ 2360.197096] ? hrtimer_interrupt+0x128/0x2c0 [ 2360.201090] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2360.204002] ? apic_timer_interrupt+0xf/0x20 [ 2360.210733] [ 2360.211848] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2360.219312] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2360.225577] ? dump_stack+0xf6/0x10e [ 2360.228435] do_raw_spin_lock+0x177/0x190 [ 2360.230988] _raw_spin_lock+0x21/0x30 [ 2360.235105] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2360.238909] obd_mod_free+0x46/0x140 [ptlrpc] [ 2360.242954] mdc_commit_open+0x91/0xb0 [mdc] [ 2360.248691] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2360.259387] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2360.265380] after_reply+0xe2c/0x1780 [ptlrpc] [ 2360.271177] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2360.279921] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2360.284126] ? do_wait_intr+0xf0/0xf0 [ 2360.286465] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2360.298492] kthread+0x1d1/0x200 [ 2360.303308] ? set_kthread_struct+0x70/0x70 [ 2360.306310] ret_from_fork+0x1f/0x30 [ 2363.874176] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 338s! [ 2363.879164] Showing busy workqueues and worker pools: [ 2363.885987] workqueue events: flags=0x0 [ 2363.889898] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2363.901720] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2363.906166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2363.919884] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2363.927703] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2363.935360] pending: kfree_rcu_monitor [ 2363.941924] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2363.949853] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2363.961207] workqueue events_long: flags=0x0 [ 2363.964135] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2363.970730] pending: lu_objects_flush [obdclass] [ 2363.979956] workqueue events_unbound: flags=0x2 [ 2363.988970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2363.993942] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2364.004709] workqueue events_power_efficient: flags=0x80 [ 2364.007489] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2364.016945] pending: neigh_periodic_work, check_lifetime [ 2364.019981] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2364.026145] pending: do_cache_clean, neigh_periodic_work [ 2364.031788] workqueue mm_percpu_wq: flags=0x8 [ 2364.034768] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2364.043503] pending: lru_add_drain_per_cpu [ 2364.049578] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2364.055702] pending: vmstat_update, lru_add_drain_per_cpu [ 2364.060608] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2364.067665] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2364.072687] workqueue writeback: flags=0x4a [ 2364.079562] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2364.084761] pending: wb_workfn, wb_workfn [ 2364.092678] workqueue xprtiod: flags=0xa [ 2364.096039] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2364.099755] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2364.107215] workqueue ipv6_addrconf: flags=0x40008 [ 2364.116553] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2364.123190] pending: addrconf_verify_work [ 2364.131038] workqueue knbd0-recv: flags=0x1a [ 2364.135756] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2364.146881] in-flight: 655:recv_work [ 2364.151021] workqueue ptlrpc_pinger: flags=0x2 [ 2364.155331] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2364.160109] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2364.164875] pool 8: cpus=0-3 flags=0x4 nice=0 hung=339s workers=4 idle: 36396 38 11188 [ 2364.169795] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2354s workers=2 idle: 658 [ 2368.058458] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2368.068455] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2368.119834] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2368.127608] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2368.133276] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2368.143346] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2368.167592] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2368.177892] RAX: 0000000000003c83 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2368.182690] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2368.194836] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2368.208334] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2368.217612] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2368.223276] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2368.240037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2368.249903] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2368.259778] Call Trace: [ 2368.262835] [ 2368.270338] ? show_regs.cold.9+0x22/0x2f [ 2368.273586] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2368.279558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2368.282566] ? watchdog+0x40/0x40 [ 2368.290979] ? hrtimer_interrupt+0x128/0x2c0 [ 2368.296046] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2368.299782] ? apic_timer_interrupt+0xf/0x20 [ 2368.302716] [ 2368.304325] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2368.311663] do_raw_spin_lock+0x177/0x190 [ 2368.314144] _raw_spin_lock+0x21/0x30 [ 2368.316406] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2368.320673] ? do_raw_spin_unlock+0x75/0x190 [ 2368.323960] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2368.331558] ? do_wait_intr+0xf0/0xf0 [ 2368.334469] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2368.337628] mdc_close+0x53b/0xd70 [mdc] [ 2368.340450] lmv_close+0x18d/0x490 [lmv] [ 2368.343279] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2368.347440] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2368.351494] ll_md_close+0x274/0x800 [lustre] [ 2368.354593] ll_file_release+0x128/0xc50 [lustre] [ 2368.357825] __fput+0x10d/0x440 [ 2368.360898] ____fput+0x12/0x20 [ 2368.365558] task_work_run+0xc6/0x110 [ 2368.370641] exit_to_usermode_loop+0x1dd/0x1f0 [ 2368.377166] do_syscall_64+0x3d6/0x440 [ 2368.379801] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2368.383723] RIP: 0033:0x7fdc0afbfcd8 [ 2368.385560] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2368.396892] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2368.401521] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2368.405627] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2368.409636] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2368.416416] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2368.424481] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2380.052008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2380.055571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2380.078154] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2380.083766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2380.090769] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2380.096483] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2380.111089] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2380.132093] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2380.141031] RDX: 0000000000003247 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2380.150360] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2380.155027] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2380.161754] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2380.168165] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2380.178375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2380.183966] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2380.188487] Call Trace: [ 2380.191414] [ 2380.192464] ? show_regs.cold.9+0x22/0x2f [ 2380.193882] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2380.197260] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2380.198827] ? watchdog+0x40/0x40 [ 2380.201293] ? hrtimer_interrupt+0x128/0x2c0 [ 2380.202776] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2380.206087] ? apic_timer_interrupt+0xf/0x20 [ 2380.211360] [ 2380.212567] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2380.221764] do_raw_spin_lock+0x177/0x190 [ 2380.230953] _raw_spin_lock+0x21/0x30 [ 2380.232488] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2380.235474] ? do_raw_spin_unlock+0x75/0x190 [ 2380.246375] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2380.251028] ? do_wait_intr+0xf0/0xf0 [ 2380.255330] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2380.260644] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2380.263556] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2380.266187] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2380.268402] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2380.271172] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2380.273555] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2380.276906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2380.279439] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2380.282240] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2380.284696] lmv_intent_open+0x338/0x1150 [lmv] [ 2380.287056] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2380.294093] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2380.299812] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2380.302687] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2380.306293] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2380.315339] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2380.317928] ? do_raw_spin_unlock+0x75/0x190 [ 2380.319833] ? _raw_spin_unlock+0x12/0x30 [ 2380.322476] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2380.327110] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2380.332406] ll_atomic_open+0x36a/0xd70 [lustre] [ 2380.334841] lookup_open+0x356/0xb40 [ 2380.336403] do_last+0x313/0xfc0 [ 2380.338297] ? nd_jump_root+0xe5/0x160 [ 2380.340095] ? path_init+0x437/0x520 [ 2380.342066] path_openat+0xf7/0x500 [ 2380.346462] ? do_raw_spin_unlock+0x75/0x190 [ 2380.348251] do_filp_open+0x99/0x140 [ 2380.350456] ? getname_flags+0x6e/0x330 [ 2380.353559] ? __check_object_size+0xff/0x256 [ 2380.356551] ? do_raw_spin_unlock+0x75/0x190 [ 2380.358438] ? _raw_spin_unlock+0x12/0x30 [ 2380.360267] do_sys_openat2+0x2b4/0x410 [ 2380.365449] do_sys_open+0x73/0xa0 [ 2380.368431] __x64_sys_openat+0x24/0x30 [ 2380.370752] do_syscall_64+0xc1/0x440 [ 2380.372693] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2380.374462] RIP: 0033:0x7fdc0afbf332 [ 2380.376933] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2380.390383] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2380.398514] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2380.412811] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2380.415093] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2380.426623] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2380.430374] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2384.051034] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 2384.055484] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2384.103587] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2384.124505] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2384.138760] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2384.148045] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2384.149712] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2384.174613] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2384.186693] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2384.189489] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2384.194574] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2384.199059] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2384.202450] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2384.217357] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2384.233007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2384.243172] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2384.248257] Call Trace: [ 2384.253001] [ 2384.253800] ? show_regs.cold.9+0x22/0x2f [ 2384.264975] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2384.268904] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2384.270933] ? watchdog+0x40/0x40 [ 2384.273302] ? hrtimer_interrupt+0x128/0x2c0 [ 2384.280476] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2384.285057] ? apic_timer_interrupt+0xf/0x20 [ 2384.291465] [ 2384.294010] ? native_safe_halt+0x2e/0x50 [ 2384.302180] kvm_wait+0xcd/0xe0 [ 2384.303274] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2384.310942] do_raw_spin_lock+0x177/0x190 [ 2384.314516] _raw_spin_lock+0x21/0x30 [ 2384.322186] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2384.331254] process_one_work+0x2c8/0x700 [ 2384.337552] worker_thread+0x296/0x6e0 [ 2384.341647] ? rescuer_thread+0x570/0x570 [ 2384.343214] kthread+0x1d1/0x200 [ 2384.347072] ? set_kthread_struct+0x70/0x70 [ 2384.349677] ret_from_fork+0x1f/0x30 [ 2392.048018] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2392.052708] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2392.093225] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2392.102447] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2392.106470] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2392.110959] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2392.121022] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2392.126175] RAX: 0000000000005b89 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2392.130051] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2392.135738] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2392.141545] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2392.146107] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2392.149511] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2392.153803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2392.156703] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2392.160314] Call Trace: [ 2392.161326] [ 2392.162812] ? show_regs.cold.9+0x22/0x2f [ 2392.166396] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2392.171933] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2392.174931] ? watchdog+0x40/0x40 [ 2392.176434] ? hrtimer_interrupt+0x128/0x2c0 [ 2392.179283] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2392.182012] ? apic_timer_interrupt+0xf/0x20 [ 2392.184207] [ 2392.185449] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2392.190136] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2392.193886] ? dump_stack+0xf6/0x10e [ 2392.199001] do_raw_spin_lock+0x177/0x190 [ 2392.202571] _raw_spin_lock+0x21/0x30 [ 2392.206265] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2392.211454] obd_mod_free+0x46/0x140 [ptlrpc] [ 2392.214577] mdc_commit_open+0x91/0xb0 [mdc] [ 2392.222490] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2392.228038] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2392.231648] after_reply+0xe2c/0x1780 [ptlrpc] [ 2392.234884] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2392.237871] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2392.240688] ? do_wait_intr+0xf0/0xf0 [ 2392.242924] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2392.246034] kthread+0x1d1/0x200 [ 2392.247634] ? set_kthread_struct+0x70/0x70 [ 2392.250674] ret_from_fork+0x1f/0x30 [ 2394.597361] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 369s! [ 2394.607236] Showing busy workqueues and worker pools: [ 2394.613939] workqueue events: flags=0x0 [ 2394.615652] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2394.619422] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2394.626909] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2394.631043] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2394.636382] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2394.639228] pending: kfree_rcu_monitor [ 2394.641115] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2394.644071] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2394.649467] workqueue events_long: flags=0x0 [ 2394.652669] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2394.655937] pending: lu_objects_flush [obdclass] [ 2394.658672] workqueue events_unbound: flags=0x2 [ 2394.660678] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2394.663301] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2394.668422] workqueue events_power_efficient: flags=0x80 [ 2394.671148] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2394.674297] pending: neigh_periodic_work, check_lifetime [ 2394.680756] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2394.686797] pending: do_cache_clean, neigh_periodic_work [ 2394.688786] workqueue mm_percpu_wq: flags=0x8 [ 2394.690966] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2394.693770] pending: lru_add_drain_per_cpu [ 2394.700642] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2394.704495] pending: vmstat_update, lru_add_drain_per_cpu [ 2394.706764] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2394.711380] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2394.713508] workqueue writeback: flags=0x4a [ 2394.715194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2394.717219] pending: wb_workfn, wb_workfn [ 2394.719381] workqueue xprtiod: flags=0xa [ 2394.721278] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2394.723105] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2394.724507] workqueue ipv6_addrconf: flags=0x40008 [ 2394.726490] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2394.729150] pending: addrconf_verify_work [ 2394.730876] workqueue knbd0-recv: flags=0x1a [ 2394.732468] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2394.734725] in-flight: 655:recv_work [ 2394.736466] workqueue ptlrpc_pinger: flags=0x2 [ 2394.742296] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2394.746103] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2394.750289] pool 8: cpus=0-3 flags=0x4 nice=0 hung=369s workers=4 idle: 36396 38 11188 [ 2394.752756] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2385s workers=2 idle: 658 [ 2396.055016] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2396.060149] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2396.093894] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2396.103995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2396.108688] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2396.111469] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2396.121990] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2396.124780] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2396.127590] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2396.131946] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2396.136735] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2396.140185] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2396.145309] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2396.148828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2396.151441] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2396.154490] Call Trace: [ 2396.155753] [ 2396.156709] ? show_regs.cold.9+0x22/0x2f [ 2396.158665] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2396.160878] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2396.162674] ? watchdog+0x40/0x40 [ 2396.164839] ? hrtimer_interrupt+0x128/0x2c0 [ 2396.170037] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2396.173630] ? apic_timer_interrupt+0xf/0x20 [ 2396.177452] [ 2396.178455] ? native_safe_halt+0x2e/0x50 [ 2396.179677] kvm_wait+0xcd/0xe0 [ 2396.182471] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2396.185643] do_raw_spin_lock+0x177/0x190 [ 2396.187702] _raw_spin_lock+0x21/0x30 [ 2396.189756] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2396.192289] ? do_raw_spin_unlock+0x75/0x190 [ 2396.193794] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2396.196366] ? do_wait_intr+0xf0/0xf0 [ 2396.197908] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2396.208049] mdc_close+0x53b/0xd70 [mdc] [ 2396.210420] lmv_close+0x18d/0x490 [lmv] [ 2396.213662] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2396.217957] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2396.223458] ll_md_close+0x274/0x800 [lustre] [ 2396.226805] ll_file_release+0x128/0xc50 [lustre] [ 2396.230697] __fput+0x10d/0x440 [ 2396.233518] ____fput+0x12/0x20 [ 2396.236362] task_work_run+0xc6/0x110 [ 2396.240617] exit_to_usermode_loop+0x1dd/0x1f0 [ 2396.242740] do_syscall_64+0x3d6/0x440 [ 2396.244913] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2396.246971] RIP: 0033:0x7fdc0afbfcd8 [ 2396.249366] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2396.257487] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2396.264679] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2396.269283] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2396.273337] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2396.276977] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2396.280014] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2408.053575] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2408.059616] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2408.119877] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2408.135402] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2408.145303] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2408.148218] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2408.172883] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2408.182020] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2408.190822] RDX: 0000000000002a83 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2408.195713] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2408.205228] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2408.210638] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2408.216207] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2408.227114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2408.229708] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2408.235608] Call Trace: [ 2408.236412] [ 2408.237018] ? show_regs.cold.9+0x22/0x2f [ 2408.245980] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2408.248333] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2408.251736] ? watchdog+0x40/0x40 [ 2408.252988] ? hrtimer_interrupt+0x128/0x2c0 [ 2408.255945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2408.258079] ? apic_timer_interrupt+0xf/0x20 [ 2408.261494] [ 2408.268344] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2408.271967] do_raw_spin_lock+0x177/0x190 [ 2408.273496] _raw_spin_lock+0x21/0x30 [ 2408.274934] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2408.277740] ? do_raw_spin_unlock+0x75/0x190 [ 2408.279479] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2408.281116] ? do_wait_intr+0xf0/0xf0 [ 2408.282208] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2408.284090] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2408.286271] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2408.288630] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2408.291877] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2408.295064] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2408.296305] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2408.298172] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2408.301915] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2408.304554] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2408.306555] lmv_intent_open+0x338/0x1150 [lmv] [ 2408.308243] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2408.310232] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2408.325252] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2408.328363] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2408.336257] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2408.338580] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2408.347878] ? do_raw_spin_unlock+0x75/0x190 [ 2408.352860] ? _raw_spin_unlock+0x12/0x30 [ 2408.354992] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2408.356941] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2408.362250] ll_atomic_open+0x36a/0xd70 [lustre] [ 2408.365503] lookup_open+0x356/0xb40 [ 2408.367709] do_last+0x313/0xfc0 [ 2408.370064] ? nd_jump_root+0xe5/0x160 [ 2408.374015] ? path_init+0x437/0x520 [ 2408.375542] path_openat+0xf7/0x500 [ 2408.381011] ? do_raw_spin_unlock+0x75/0x190 [ 2408.388181] do_filp_open+0x99/0x140 [ 2408.391567] ? getname_flags+0x6e/0x330 [ 2408.395271] ? __check_object_size+0xff/0x256 [ 2408.401846] ? do_raw_spin_unlock+0x75/0x190 [ 2408.405421] ? _raw_spin_unlock+0x12/0x30 [ 2408.410729] do_sys_openat2+0x2b4/0x410 [ 2408.414771] do_sys_open+0x73/0xa0 [ 2408.423189] __x64_sys_openat+0x24/0x30 [ 2408.424532] do_syscall_64+0xc1/0x440 [ 2408.434515] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2408.436644] RIP: 0033:0x7fdc0afbf332 [ 2408.438962] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2408.455051] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2408.460904] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2408.464883] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2408.468044] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2408.471861] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2408.474819] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2412.060118] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2412.073659] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2412.114029] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2412.119487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2412.123784] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2412.130415] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2412.134583] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2412.144090] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2412.147466] RAX: 0000000000007915 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2412.151607] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2412.158684] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2412.163969] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2412.168697] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2412.170843] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2412.173134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2412.174695] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2412.176941] Call Trace: [ 2412.177810] [ 2412.178372] ? show_regs.cold.9+0x22/0x2f [ 2412.180361] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2412.181937] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2412.184896] ? watchdog+0x40/0x40 [ 2412.185882] ? hrtimer_interrupt+0x128/0x2c0 [ 2412.187739] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2412.188978] ? apic_timer_interrupt+0xf/0x20 [ 2412.190121] [ 2412.190674] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2412.192753] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2412.194767] do_raw_spin_lock+0x177/0x190 [ 2412.196339] _raw_spin_lock+0x21/0x30 [ 2412.197531] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2412.203908] process_one_work+0x2c8/0x700 [ 2412.206768] worker_thread+0x296/0x6e0 [ 2412.213827] ? rescuer_thread+0x570/0x570 [ 2412.215171] kthread+0x1d1/0x200 [ 2412.216191] ? set_kthread_struct+0x70/0x70 [ 2412.218685] ret_from_fork+0x1f/0x30 [ 2420.048110] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2420.051311] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2420.078924] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2420.083372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2420.087601] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2420.090237] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2420.098608] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2420.102027] RAX: 0000000000001589 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2420.105347] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2420.108617] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2420.111603] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2420.114874] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2420.117138] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2420.119609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2420.121984] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2420.124957] Call Trace: [ 2420.125927] [ 2420.126672] ? show_regs.cold.9+0x22/0x2f [ 2420.128626] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2420.130866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2420.132670] ? watchdog+0x40/0x40 [ 2420.133828] ? hrtimer_interrupt+0x128/0x2c0 [ 2420.135616] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2420.137000] ? apic_timer_interrupt+0xf/0x20 [ 2420.139526] [ 2420.141068] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2420.143188] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2420.145713] ? dump_stack+0xf6/0x10e [ 2420.147578] do_raw_spin_lock+0x177/0x190 [ 2420.149493] _raw_spin_lock+0x21/0x30 [ 2420.151254] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2420.154803] obd_mod_free+0x46/0x140 [ptlrpc] [ 2420.157465] mdc_commit_open+0x91/0xb0 [mdc] [ 2420.158769] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2420.160831] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2420.163397] after_reply+0xe2c/0x1780 [ptlrpc] [ 2420.164821] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2420.167626] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2420.169866] ? do_wait_intr+0xf0/0xf0 [ 2420.171708] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2420.174589] kthread+0x1d1/0x200 [ 2420.175554] ? set_kthread_struct+0x70/0x70 [ 2420.178622] ret_from_fork+0x1f/0x30 [ 2424.054496] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2424.066841] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2424.097880] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2424.104143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2424.114178] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2424.120835] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2424.139905] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2424.144628] RAX: 0000000000007f94 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2424.148758] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2424.153492] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2424.157815] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2424.161851] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2424.165842] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2424.170245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2424.173405] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2424.177287] Call Trace: [ 2424.178761] [ 2424.179359] ? show_regs.cold.9+0x22/0x2f [ 2424.180862] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2424.183147] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2424.186075] ? watchdog+0x40/0x40 [ 2424.187853] ? hrtimer_interrupt+0x128/0x2c0 [ 2424.189888] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2424.191329] ? apic_timer_interrupt+0xf/0x20 [ 2424.193243] [ 2424.194120] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2424.196226] do_raw_spin_lock+0x177/0x190 [ 2424.197889] _raw_spin_lock+0x21/0x30 [ 2424.198870] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2424.201114] ? do_raw_spin_unlock+0x75/0x190 [ 2424.202303] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2424.204855] ? do_wait_intr+0xf0/0xf0 [ 2424.206143] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2424.208659] mdc_close+0x53b/0xd70 [mdc] [ 2424.209997] lmv_close+0x18d/0x490 [lmv] [ 2424.211226] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2424.213148] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2424.214813] ll_md_close+0x274/0x800 [lustre] [ 2424.216243] ll_file_release+0x128/0xc50 [lustre] [ 2424.217872] __fput+0x10d/0x440 [ 2424.218817] ____fput+0x12/0x20 [ 2424.219855] task_work_run+0xc6/0x110 [ 2424.221062] exit_to_usermode_loop+0x1dd/0x1f0 [ 2424.222547] do_syscall_64+0x3d6/0x440 [ 2424.223840] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2424.225669] RIP: 0033:0x7fdc0afbfcd8 [ 2424.226790] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2424.233046] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2424.237072] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2424.240802] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2424.244829] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2424.248143] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2424.251438] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2424.808144] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 399s! [ 2424.819769] Showing busy workqueues and worker pools: [ 2424.822802] workqueue events: flags=0x0 [ 2424.824779] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2424.828486] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2424.843459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2424.856196] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2424.870361] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2424.880664] pending: kfree_rcu_monitor [ 2424.883468] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2424.890928] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2424.898366] workqueue events_long: flags=0x0 [ 2424.900772] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2424.906603] pending: lu_objects_flush [obdclass] [ 2424.909874] workqueue events_unbound: flags=0x2 [ 2424.912313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2424.916915] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2424.926498] workqueue events_power_efficient: flags=0x80 [ 2424.929345] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2424.933131] pending: neigh_periodic_work, check_lifetime [ 2424.935616] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2424.939780] pending: do_cache_clean, neigh_periodic_work [ 2424.942809] workqueue mm_percpu_wq: flags=0x8 [ 2424.944718] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2424.947670] pending: lru_add_drain_per_cpu [ 2424.951302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2424.955336] pending: vmstat_update, lru_add_drain_per_cpu [ 2424.959924] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2424.964594] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2424.967480] workqueue writeback: flags=0x4a [ 2424.968909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2424.973660] pending: wb_workfn, wb_workfn [ 2424.977795] workqueue xprtiod: flags=0xa [ 2424.981162] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2424.986795] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2424.994048] workqueue ipv6_addrconf: flags=0x40008 [ 2424.999851] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2425.003611] pending: addrconf_verify_work [ 2425.006544] workqueue knbd0-recv: flags=0x1a [ 2425.008551] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2425.010929] in-flight: 655:recv_work [ 2425.012807] workqueue ptlrpc_pinger: flags=0x2 [ 2425.014897] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2425.017830] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2425.021981] pool 8: cpus=0-3 flags=0x4 nice=0 hung=400s workers=4 idle: 36396 38 11188 [ 2425.027207] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2415s workers=2 idle: 658 [ 2436.052065] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2436.056365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2436.080991] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2436.093437] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2436.101259] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2436.104666] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2436.120745] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2436.125723] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2436.149731] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2436.163675] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2436.183646] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2436.201774] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2436.212266] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2436.223282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2436.227653] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2436.234836] Call Trace: [ 2436.236606] [ 2436.237372] ? show_regs.cold.9+0x22/0x2f [ 2436.238651] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2436.243261] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2436.247081] ? watchdog+0x40/0x40 [ 2436.249343] ? hrtimer_interrupt+0x128/0x2c0 [ 2436.255653] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2436.259726] ? apic_timer_interrupt+0xf/0x20 [ 2436.263896] [ 2436.265941] ? native_safe_halt+0x2e/0x50 [ 2436.279683] kvm_wait+0xcd/0xe0 [ 2436.287044] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2436.292547] do_raw_spin_lock+0x177/0x190 [ 2436.295340] _raw_spin_lock+0x21/0x30 [ 2436.296788] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2436.300239] ? do_raw_spin_unlock+0x75/0x190 [ 2436.303138] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2436.305338] ? do_wait_intr+0xf0/0xf0 [ 2436.308041] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2436.313152] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2436.316877] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2436.322740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2436.328096] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2436.334807] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2436.337281] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2436.343296] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2436.350351] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2436.354497] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2436.356585] lmv_intent_open+0x338/0x1150 [lmv] [ 2436.360156] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2436.362057] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2436.372045] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2436.375030] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2436.382816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2436.395034] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2436.403594] ? do_raw_spin_unlock+0x75/0x190 [ 2436.406726] ? _raw_spin_unlock+0x12/0x30 [ 2436.411188] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2436.415374] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2436.420039] ll_atomic_open+0x36a/0xd70 [lustre] [ 2436.430842] lookup_open+0x356/0xb40 [ 2436.435045] do_last+0x313/0xfc0 [ 2436.436256] ? nd_jump_root+0xe5/0x160 [ 2436.442222] ? path_init+0x437/0x520 [ 2436.443483] path_openat+0xf7/0x500 [ 2436.444745] ? do_raw_spin_unlock+0x75/0x190 [ 2436.446662] do_filp_open+0x99/0x140 [ 2436.447968] ? getname_flags+0x6e/0x330 [ 2436.450140] ? __check_object_size+0xff/0x256 [ 2436.455078] ? do_raw_spin_unlock+0x75/0x190 [ 2436.457365] ? _raw_spin_unlock+0x12/0x30 [ 2436.459289] do_sys_openat2+0x2b4/0x410 [ 2436.461629] do_sys_open+0x73/0xa0 [ 2436.463053] __x64_sys_openat+0x24/0x30 [ 2436.467437] do_syscall_64+0xc1/0x440 [ 2436.468972] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2436.471040] RIP: 0033:0x7fdc0afbf332 [ 2436.472643] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2436.481846] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2436.486851] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2436.498898] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2436.503498] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2436.509139] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2436.514202] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2439.354022] rcu: INFO: rcu_sched self-detected stall on CPU [ 2439.361253] rcu: 0-...!: (251852 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 2439.375276] (t=420011 jiffies g=270761 q=2738) [ 2439.377255] rcu: rcu_sched kthread starved for 414943 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 2439.384133] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2439.387412] rcu: RCU grace-period kthread stack dump: [ 2439.389241] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2439.394307] Call Trace: [ 2439.395028] __schedule+0x351/0xcb0 [ 2439.396302] schedule+0xc0/0x180 [ 2439.397707] schedule_timeout+0xb4/0x190 [ 2439.399313] ? do_raw_spin_unlock+0x75/0x190 [ 2439.400526] ? __next_timer_interrupt+0x160/0x160 [ 2439.402236] ? prepare_to_swait_event+0xf3/0x140 [ 2439.403546] rcu_gp_kthread+0x8dd/0x15b0 [ 2439.410225] ? force_qs_rnp+0x2c0/0x2c0 [ 2439.415260] kthread+0x1d1/0x200 [ 2439.416397] ? set_kthread_struct+0x70/0x70 [ 2439.420621] ret_from_fork+0x1f/0x30 [ 2439.422230] rcu: Stack dump where RCU GP kthread last ran: [ 2439.425289] Sending NMI from CPU 0 to CPUs 1: [ 2439.429117] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 2439.430552] NMI backtrace for cpu 0 [ 2439.440206] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2439.446318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2439.451572] Call Trace: [ 2439.452229] [ 2439.452764] ? dump_stack+0xbb/0x10e [ 2439.458139] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2439.459980] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2439.461674] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2439.465264] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2439.467584] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2439.470373] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2439.476233] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2439.479162] ? update_process_times+0x6d/0xc0 [ 2439.482197] ? tick_sched_handle.isra.18+0x27/0x90 [ 2439.484604] ? tick_sched_timer+0x53/0xb0 [ 2439.486450] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2439.489313] ? tick_sched_handle.isra.18+0x90/0x90 [ 2439.492909] ? hrtimer_interrupt+0x128/0x2c0 [ 2439.494631] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2439.496430] ? apic_timer_interrupt+0xf/0x20 [ 2439.499458] [ 2439.501019] ? native_safe_halt+0x2e/0x50 [ 2439.504652] ? kvm_wait+0xcd/0xe0 [ 2439.507288] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2439.510671] ? dump_stack+0xf6/0x10e [ 2439.512117] ? do_raw_spin_lock+0x177/0x190 [ 2439.513689] ? _raw_spin_lock+0x21/0x30 [ 2439.515309] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2439.518356] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2439.520435] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2439.523174] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2439.526232] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2439.529656] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2439.534471] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2439.541208] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2439.546558] ? do_wait_intr+0xf0/0xf0 [ 2439.550328] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2439.557403] ? kthread+0x1d1/0x200 [ 2439.560199] ? set_kthread_struct+0x70/0x70 [ 2439.564170] ? ret_from_fork+0x1f/0x30 [ 2452.058413] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2452.072537] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2452.134466] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2452.151290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2452.166604] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2452.173285] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2452.198439] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2452.210614] RAX: 0000000000006b3b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2452.223528] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2452.230748] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2452.237826] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2452.247863] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2452.259950] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2452.272263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2452.278373] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2452.285512] Call Trace: [ 2452.292019] [ 2452.294830] ? show_regs.cold.9+0x22/0x2f [ 2452.300323] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2452.307752] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2452.312187] ? watchdog+0x40/0x40 [ 2452.319848] ? hrtimer_interrupt+0x128/0x2c0 [ 2452.324375] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2452.332225] ? apic_timer_interrupt+0xf/0x20 [ 2452.333712] [ 2452.334340] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2452.342422] do_raw_spin_lock+0x177/0x190 [ 2452.345576] _raw_spin_lock+0x21/0x30 [ 2452.354219] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2452.367463] ? do_raw_spin_unlock+0x75/0x190 [ 2452.369728] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2452.378212] ? do_wait_intr+0xf0/0xf0 [ 2452.384437] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2452.391368] mdc_close+0x53b/0xd70 [mdc] [ 2452.393508] lmv_close+0x18d/0x490 [lmv] [ 2452.399724] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2452.404196] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2452.408109] ll_md_close+0x274/0x800 [lustre] [ 2452.414428] ll_file_release+0x128/0xc50 [lustre] [ 2452.419557] __fput+0x10d/0x440 [ 2452.421552] ____fput+0x12/0x20 [ 2452.422884] task_work_run+0xc6/0x110 [ 2452.424232] exit_to_usermode_loop+0x1dd/0x1f0 [ 2452.425873] do_syscall_64+0x3d6/0x440 [ 2452.426985] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2452.429165] RIP: 0033:0x7fdc0afbfcd8 [ 2452.430566] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2452.438864] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2452.442155] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2452.445081] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2452.453664] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2452.457638] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2452.460400] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2455.527580] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 430s! [ 2455.540284] Showing busy workqueues and worker pools: [ 2455.544107] workqueue events: flags=0x0 [ 2455.549049] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2455.556412] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2455.562763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2455.568003] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2455.583280] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2455.595766] pending: kfree_rcu_monitor [ 2455.599980] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2455.611903] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2455.625262] workqueue events_long: flags=0x0 [ 2455.629908] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2455.633978] pending: lu_objects_flush [obdclass] [ 2455.643937] workqueue events_unbound: flags=0x2 [ 2455.649330] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2455.656805] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2455.667011] workqueue events_power_efficient: flags=0x80 [ 2455.673752] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2455.682439] pending: neigh_periodic_work, check_lifetime [ 2455.688227] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2455.693611] pending: do_cache_clean, neigh_periodic_work [ 2455.696687] workqueue mm_percpu_wq: flags=0x8 [ 2455.698217] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2455.706208] pending: lru_add_drain_per_cpu [ 2455.716030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2455.723182] pending: vmstat_update, lru_add_drain_per_cpu [ 2455.726132] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2455.735063] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2455.739426] workqueue writeback: flags=0x4a [ 2455.746023] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2455.752403] pending: wb_workfn, wb_workfn [ 2455.755616] workqueue xprtiod: flags=0xa [ 2455.759882] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2455.768429] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2455.774578] workqueue ipv6_addrconf: flags=0x40008 [ 2455.784408] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2455.794848] pending: addrconf_verify_work [ 2455.798841] workqueue knbd0-recv: flags=0x1a [ 2455.800689] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2455.811946] in-flight: 655:recv_work [ 2455.819280] workqueue ptlrpc_pinger: flags=0x2 [ 2455.821381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2455.838192] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2455.847114] pool 8: cpus=0-3 flags=0x4 nice=0 hung=430s workers=4 idle: 36396 38 11188 [ 2455.860817] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2446s workers=2 idle: 658 [ 2464.051127] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2464.052086] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2464.058229] Modules linked in: [ 2464.061214] Modules linked in: [ 2464.061222] lustre(O) [ 2464.062538] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2464.062620] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2464.064583] osp(O) [ 2464.065610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2464.113426] ofd(O) [ 2464.117240] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2464.117264] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2464.117268] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2464.117274] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2464.117277] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2464.117279] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2464.117281] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2464.117284] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2464.117287] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2464.117289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2464.117292] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2464.117298] Call Trace: [ 2464.117303] [ 2464.117308] ? show_regs.cold.9+0x22/0x2f [ 2464.117316] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2464.117322] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2464.117391] ? watchdog+0x40/0x40 [ 2464.117396] ? hrtimer_interrupt+0x128/0x2c0 [ 2464.117400] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2464.117406] ? apic_timer_interrupt+0xf/0x20 [ 2464.117410] [ 2464.117414] ? native_safe_halt+0x2e/0x50 [ 2464.117417] kvm_wait+0xcd/0xe0 [ 2464.117424] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2464.117431] do_raw_spin_lock+0x177/0x190 [ 2464.117435] _raw_spin_lock+0x21/0x30 [ 2464.117441] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2464.117840] ? do_raw_spin_unlock+0x75/0x190 [ 2464.117846] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2464.118111] ? do_wait_intr+0xf0/0xf0 [ 2464.118120] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2464.118399] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2464.118663] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2464.118891] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2464.119011] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2464.121275] lod(O) [ 2464.125741] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2464.126836] mdt(O) [ 2464.129106] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2464.146564] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2464.146636] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2464.146643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2464.146649] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2464.146966] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2464.146975] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2464.146979] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2464.146983] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2464.146986] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2464.146988] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2464.146990] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2464.146993] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2464.146996] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2464.146999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2464.147001] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2464.147010] Call Trace: [ 2464.147020] [ 2464.147035] ? show_regs.cold.9+0x22/0x2f [ 2464.147046] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2464.147058] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2464.147063] ? watchdog+0x40/0x40 [ 2464.147068] ? hrtimer_interrupt+0x128/0x2c0 [ 2464.147073] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2464.147079] ? apic_timer_interrupt+0xf/0x20 [ 2464.147084] [ 2464.147087] ? native_safe_halt+0x2e/0x50 [ 2464.147091] kvm_wait+0xcd/0xe0 [ 2464.147099] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2464.147107] do_raw_spin_lock+0x177/0x190 [ 2464.147112] _raw_spin_lock+0x21/0x30 [ 2464.147116] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2464.147439] process_one_work+0x2c8/0x700 [ 2464.147446] worker_thread+0x296/0x6e0 [ 2464.147450] ? rescuer_thread+0x570/0x570 [ 2464.147454] kthread+0x1d1/0x200 [ 2464.147458] ? set_kthread_struct+0x70/0x70 [ 2464.147461] ret_from_fork+0x1f/0x30 [ 2464.305878] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2464.307966] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2464.310274] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2464.312482] lmv_intent_open+0x338/0x1150 [lmv] [ 2464.314095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2464.316137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2464.318803] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2464.320702] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2464.322039] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2464.324067] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2464.326276] ? do_raw_spin_unlock+0x75/0x190 [ 2464.329099] ? _raw_spin_unlock+0x12/0x30 [ 2464.330821] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2464.332737] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2464.334580] ll_atomic_open+0x36a/0xd70 [lustre] [ 2464.336286] lookup_open+0x356/0xb40 [ 2464.337971] do_last+0x313/0xfc0 [ 2464.339138] ? nd_jump_root+0xe5/0x160 [ 2464.340451] ? path_init+0x437/0x520 [ 2464.341978] path_openat+0xf7/0x500 [ 2464.343284] ? do_raw_spin_unlock+0x75/0x190 [ 2464.345176] do_filp_open+0x99/0x140 [ 2464.346429] ? getname_flags+0x6e/0x330 [ 2464.351130] ? __check_object_size+0xff/0x256 [ 2464.353905] ? do_raw_spin_unlock+0x75/0x190 [ 2464.358053] ? _raw_spin_unlock+0x12/0x30 [ 2464.360721] do_sys_openat2+0x2b4/0x410 [ 2464.363915] do_sys_open+0x73/0xa0 [ 2464.365183] __x64_sys_openat+0x24/0x30 [ 2464.369717] do_syscall_64+0xc1/0x440 [ 2464.372537] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2464.376702] RIP: 0033:0x7fdc0afbf332 [ 2464.379545] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2464.389461] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2464.396288] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2464.399097] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2464.401891] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2464.409254] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2464.415411] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2480.048335] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2480.056539] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2480.062185] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2480.062263] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2480.062270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2480.062273] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2480.062288] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2480.062292] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2480.062297] RAX: 0000000000007d86 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2480.062299] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2480.062301] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2480.062303] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2480.062305] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2480.062309] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2480.062311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2480.062314] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2480.062323] Call Trace: [ 2480.062328] [ 2480.062335] ? show_regs.cold.9+0x22/0x2f [ 2480.062346] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2480.062352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2480.062356] ? watchdog+0x40/0x40 [ 2480.062361] ? hrtimer_interrupt+0x128/0x2c0 [ 2480.062365] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2480.062373] ? apic_timer_interrupt+0xf/0x20 [ 2480.062377] [ 2480.062380] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2480.062386] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2480.062390] ? dump_stack+0xf6/0x10e [ 2480.062396] do_raw_spin_lock+0x177/0x190 [ 2480.062401] _raw_spin_lock+0x21/0x30 [ 2480.062407] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2480.062703] obd_mod_free+0x46/0x140 [ptlrpc] [ 2480.062899] mdc_commit_open+0x91/0xb0 [mdc] [ 2480.062939] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2480.072048] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2480.072131] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2480.072140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2480.072143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2480.072157] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2480.072161] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2480.072165] RAX: 000000000000380a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2480.072168] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2480.072170] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2480.072172] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2480.072174] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2480.072177] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2480.072180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2480.072182] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2480.072191] Call Trace: [ 2480.072196] [ 2480.072202] ? show_regs.cold.9+0x22/0x2f [ 2480.072212] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2480.072332] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2480.072338] ? watchdog+0x40/0x40 [ 2480.072342] ? hrtimer_interrupt+0x128/0x2c0 [ 2480.072346] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2480.072354] ? apic_timer_interrupt+0xf/0x20 [ 2480.072359] [ 2480.072362] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2480.072370] do_raw_spin_lock+0x177/0x190 [ 2480.072376] _raw_spin_lock+0x21/0x30 [ 2480.072381] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2480.072697] ? do_raw_spin_unlock+0x75/0x190 [ 2480.072703] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2480.072906] ? do_wait_intr+0xf0/0xf0 [ 2480.072914] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2480.073349] mdc_close+0x53b/0xd70 [mdc] [ 2480.073468] lmv_close+0x18d/0x490 [lmv] [ 2480.073497] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2480.073628] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2480.073717] ll_md_close+0x274/0x800 [lustre] [ 2480.073812] ll_file_release+0x128/0xc50 [lustre] [ 2480.074665] __fput+0x10d/0x440 [ 2480.074675] ____fput+0x12/0x20 [ 2480.074679] task_work_run+0xc6/0x110 [ 2480.074693] exit_to_usermode_loop+0x1dd/0x1f0 [ 2480.074702] do_syscall_64+0x3d6/0x440 [ 2480.074706] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2480.074718] RIP: 0033:0x7fdc0afbfcd8 [ 2480.074727] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2480.074731] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2480.074737] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2480.074739] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2480.074741] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2480.074744] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2480.074746] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2480.655976] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2480.659161] after_reply+0xe2c/0x1780 [ptlrpc] [ 2480.661949] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2480.667278] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2480.674879] ? do_wait_intr+0xf0/0xf0 [ 2480.675995] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2480.682106] kthread+0x1d1/0x200 [ 2480.683262] ? set_kthread_struct+0x70/0x70 [ 2480.686555] ret_from_fork+0x1f/0x30 [ 2486.239063] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 461s! [ 2486.245424] Showing busy workqueues and worker pools: [ 2486.252016] workqueue events: flags=0x0 [ 2486.254432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2486.261353] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2486.270255] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2486.280398] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2486.293606] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2486.301558] pending: kfree_rcu_monitor [ 2486.304248] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2486.311555] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2486.315222] workqueue events_long: flags=0x0 [ 2486.316978] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2486.320510] pending: lu_objects_flush [obdclass] [ 2486.322752] workqueue events_unbound: flags=0x2 [ 2486.324553] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2486.326590] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2486.330837] workqueue events_power_efficient: flags=0x80 [ 2486.333122] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2486.336338] pending: neigh_periodic_work, check_lifetime [ 2486.338679] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2486.341785] pending: do_cache_clean, neigh_periodic_work [ 2486.343940] workqueue mm_percpu_wq: flags=0x8 [ 2486.349783] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2486.355594] pending: lru_add_drain_per_cpu [ 2486.359340] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2486.364588] pending: vmstat_update, lru_add_drain_per_cpu [ 2486.367208] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2486.370883] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2486.377837] workqueue writeback: flags=0x4a [ 2486.388310] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2486.393789] pending: wb_workfn, wb_workfn [ 2486.396649] workqueue xprtiod: flags=0xa [ 2486.398686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2486.402340] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2486.408404] workqueue ipv6_addrconf: flags=0x40008 [ 2486.412288] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2486.415473] pending: addrconf_verify_work [ 2486.421599] workqueue knbd0-recv: flags=0x1a [ 2486.426224] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2486.429751] in-flight: 655:recv_work [ 2486.432534] workqueue ptlrpc_pinger: flags=0x2 [ 2486.434455] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2486.439428] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2486.447497] pool 8: cpus=0-3 flags=0x4 nice=0 hung=461s workers=4 idle: 36396 38 11188 [ 2486.454620] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2477s workers=2 idle: 658 [ 2492.053099] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2492.053524] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2492.061759] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2492.062285] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2492.062292] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2492.062295] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2492.062310] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2492.062313] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2492.062317] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2492.062319] RDX: 00000000000071f2 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2492.062321] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2492.062323] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2492.062326] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2492.062329] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2492.062332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2492.062334] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2492.062343] Call Trace: [ 2492.062349] [ 2492.062355] ? show_regs.cold.9+0x22/0x2f [ 2492.062366] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2492.062373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2492.062377] ? watchdog+0x40/0x40 [ 2492.062381] ? hrtimer_interrupt+0x128/0x2c0 [ 2492.062385] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2492.062393] ? apic_timer_interrupt+0xf/0x20 [ 2492.062397] [ 2492.062400] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2492.062405] do_raw_spin_lock+0x177/0x190 [ 2492.062410] _raw_spin_lock+0x21/0x30 [ 2492.062417] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2492.062718] ? do_raw_spin_unlock+0x75/0x190 [ 2492.062728] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2492.063259] ? do_wait_intr+0xf0/0xf0 [ 2492.063269] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2492.063475] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2492.063744] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2492.064160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2492.064357] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2492.064392] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2492.064426] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2492.064457] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2492.064550] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2492.081804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2492.082033] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2492.082042] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2492.082048] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2492.082512] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2492.082523] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2492.082679] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2492.082685] RAX: 000000000000708a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2492.082688] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2492.082690] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2492.082692] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2492.082695] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2492.082698] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2492.082701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2492.082703] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2492.082711] Call Trace: [ 2492.082716] [ 2492.082721] ? show_regs.cold.9+0x22/0x2f [ 2492.082731] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2492.082738] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2492.082743] ? watchdog+0x40/0x40 [ 2492.082747] ? hrtimer_interrupt+0x128/0x2c0 [ 2492.082751] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2492.082760] ? apic_timer_interrupt+0xf/0x20 [ 2492.082765] [ 2492.082768] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2492.082774] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2492.082780] do_raw_spin_lock+0x177/0x190 [ 2492.082784] _raw_spin_lock+0x21/0x30 [ 2492.082789] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2492.084738] process_one_work+0x2c8/0x700 [ 2492.084749] worker_thread+0x296/0x6e0 [ 2492.084754] ? rescuer_thread+0x570/0x570 [ 2492.084758] kthread+0x1d1/0x200 [ 2492.084763] ? set_kthread_struct+0x70/0x70 [ 2492.084767] ret_from_fork+0x1f/0x30 [ 2492.755961] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2492.757999] lmv_intent_open+0x338/0x1150 [lmv] [ 2492.766376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2492.777577] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2492.785654] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2492.788349] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2492.791919] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2492.800736] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2492.805785] ? do_raw_spin_unlock+0x75/0x190 [ 2492.811735] ? _raw_spin_unlock+0x12/0x30 [ 2492.824729] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2492.828532] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2492.844032] ll_atomic_open+0x36a/0xd70 [lustre] [ 2492.852373] lookup_open+0x356/0xb40 [ 2492.853681] do_last+0x313/0xfc0 [ 2492.854512] ? nd_jump_root+0xe5/0x160 [ 2492.860405] ? path_init+0x437/0x520 [ 2492.861578] path_openat+0xf7/0x500 [ 2492.862500] ? do_raw_spin_unlock+0x75/0x190 [ 2492.867757] do_filp_open+0x99/0x140 [ 2492.869381] ? getname_flags+0x6e/0x330 [ 2492.873451] ? __check_object_size+0xff/0x256 [ 2492.877660] ? do_raw_spin_unlock+0x75/0x190 [ 2492.881735] ? _raw_spin_unlock+0x12/0x30 [ 2492.884254] do_sys_openat2+0x2b4/0x410 [ 2492.890166] do_sys_open+0x73/0xa0 [ 2492.894711] __x64_sys_openat+0x24/0x30 [ 2492.897255] do_syscall_64+0xc1/0x440 [ 2492.898540] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2492.900153] RIP: 0033:0x7fdc0afbf332 [ 2492.905531] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2492.925532] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2492.931571] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2492.937050] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2492.942070] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2492.945754] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2492.951767] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2508.055371] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2508.066241] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2508.096379] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2508.101674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2508.106116] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2508.109347] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2508.120002] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2508.123673] RAX: 000000000000728e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2508.126866] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2508.130171] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2508.133780] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2508.137618] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2508.141475] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2508.146419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2508.149525] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2508.153738] Call Trace: [ 2508.154952] [ 2508.156046] ? show_regs.cold.9+0x22/0x2f [ 2508.158612] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2508.161118] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2508.163628] ? watchdog+0x40/0x40 [ 2508.165987] ? hrtimer_interrupt+0x128/0x2c0 [ 2508.168428] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2508.172096] ? apic_timer_interrupt+0xf/0x20 [ 2508.175196] [ 2508.177092] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2508.180467] do_raw_spin_lock+0x177/0x190 [ 2508.183330] _raw_spin_lock+0x21/0x30 [ 2508.185953] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2508.189611] ? do_raw_spin_unlock+0x75/0x190 [ 2508.192283] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2508.195462] ? do_wait_intr+0xf0/0xf0 [ 2508.197245] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2508.200737] mdc_close+0x53b/0xd70 [mdc] [ 2508.203028] lmv_close+0x18d/0x490 [lmv] [ 2508.206177] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2508.209497] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2508.212056] ll_md_close+0x274/0x800 [lustre] [ 2508.214414] ll_file_release+0x128/0xc50 [lustre] [ 2508.216919] __fput+0x10d/0x440 [ 2508.218472] ____fput+0x12/0x20 [ 2508.220074] task_work_run+0xc6/0x110 [ 2508.221621] exit_to_usermode_loop+0x1dd/0x1f0 [ 2508.224036] do_syscall_64+0x3d6/0x440 [ 2508.226065] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2508.228633] RIP: 0033:0x7fdc0afbfcd8 [ 2508.230437] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2508.239309] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2508.243026] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2508.246772] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2508.250143] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2508.253434] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2508.257140] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2512.052358] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2512.057775] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2512.112652] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2512.122731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2512.127761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2512.135888] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2512.159765] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2512.164074] RAX: 0000000000007e09 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2512.169841] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2512.176051] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2512.183069] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2512.209949] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2512.218884] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2512.228051] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2512.232132] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2512.235951] Call Trace: [ 2512.237179] [ 2512.238220] ? show_regs.cold.9+0x22/0x2f [ 2512.240661] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2512.243092] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2512.245558] ? watchdog+0x40/0x40 [ 2512.247115] ? hrtimer_interrupt+0x128/0x2c0 [ 2512.248847] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2512.251389] ? apic_timer_interrupt+0xf/0x20 [ 2512.253921] [ 2512.254951] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2512.257760] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2512.260854] ? dump_stack+0xf6/0x10e [ 2512.262958] do_raw_spin_lock+0x177/0x190 [ 2512.264931] _raw_spin_lock+0x21/0x30 [ 2512.267060] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2512.270040] obd_mod_free+0x46/0x140 [ptlrpc] [ 2512.272407] mdc_commit_open+0x91/0xb0 [mdc] [ 2512.274703] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2512.277573] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2512.279994] after_reply+0xe2c/0x1780 [ptlrpc] [ 2512.281764] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2512.283696] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2512.285684] ? do_wait_intr+0xf0/0xf0 [ 2512.287628] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2512.289596] kthread+0x1d1/0x200 [ 2512.290607] ? set_kthread_struct+0x70/0x70 [ 2512.291969] ret_from_fork+0x1f/0x30 [ 2516.961057] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 492s! [ 2516.972559] Showing busy workqueues and worker pools: [ 2516.979399] workqueue events: flags=0x0 [ 2516.982766] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2516.993531] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2517.010032] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2517.020023] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2517.043607] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2517.056017] pending: kfree_rcu_monitor [ 2517.067524] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2517.069979] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2517.091714] workqueue events_long: flags=0x0 [ 2517.097481] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2517.106038] pending: lu_objects_flush [obdclass] [ 2517.123867] workqueue events_unbound: flags=0x2 [ 2517.130352] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2517.140457] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2517.153604] workqueue events_power_efficient: flags=0x80 [ 2517.157040] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2517.165531] pending: neigh_periodic_work, check_lifetime [ 2517.180240] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2517.189133] pending: do_cache_clean, neigh_periodic_work [ 2517.194063] workqueue mm_percpu_wq: flags=0x8 [ 2517.207405] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2517.219584] pending: lru_add_drain_per_cpu [ 2517.225973] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2517.231480] pending: vmstat_update, lru_add_drain_per_cpu [ 2517.243786] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2517.256235] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2517.266470] workqueue writeback: flags=0x4a [ 2517.270462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2517.276881] pending: wb_workfn, wb_workfn [ 2517.286977] workqueue xprtiod: flags=0xa [ 2517.291014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2517.303200] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2517.317080] workqueue ipv6_addrconf: flags=0x40008 [ 2517.324083] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2517.333355] pending: addrconf_verify_work [ 2517.342930] workqueue knbd0-recv: flags=0x1a [ 2517.348286] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2517.357048] in-flight: 655:recv_work [ 2517.361208] workqueue ptlrpc_pinger: flags=0x2 [ 2517.366527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2517.373104] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2517.389447] pool 8: cpus=0-3 flags=0x4 nice=0 hung=492s workers=4 idle: 36396 38 11188 [ 2517.405039] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2507s workers=2 idle: 658 [ 2520.052012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2520.053769] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2520.055608] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2520.066803] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2520.066816] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2520.066819] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2520.066831] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2520.066835] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2520.066841] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2520.066844] RDX: 00000000000044be RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2520.066846] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2520.066848] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2520.066850] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2520.066853] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2520.066856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2520.066858] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2520.066865] Call Trace: [ 2520.066870] [ 2520.066875] ? show_regs.cold.9+0x22/0x2f [ 2520.066882] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2520.066888] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2520.066891] ? watchdog+0x40/0x40 [ 2520.066895] ? hrtimer_interrupt+0x128/0x2c0 [ 2520.066899] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2520.066908] ? apic_timer_interrupt+0xf/0x20 [ 2520.066912] [ 2520.066915] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2520.066920] do_raw_spin_lock+0x177/0x190 [ 2520.066924] _raw_spin_lock+0x21/0x30 [ 2520.066931] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2520.067211] ? do_raw_spin_unlock+0x75/0x190 [ 2520.067216] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2520.067475] ? do_wait_intr+0xf0/0xf0 [ 2520.067483] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2520.067674] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2520.067903] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2520.068115] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2520.068231] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2520.068342] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2520.068373] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2520.068403] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2520.068490] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2520.068708] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2520.068750] lmv_intent_open+0x338/0x1150 [lmv] [ 2520.068779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2520.068868] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2520.068952] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2520.068976] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2520.069062] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2520.069145] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2520.069201] ? do_raw_spin_unlock+0x75/0x190 [ 2520.069205] ? _raw_spin_unlock+0x12/0x30 [ 2520.069210] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2520.069229] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2520.069303] ll_atomic_open+0x36a/0xd70 [lustre] [ 2520.069387] lookup_open+0x356/0xb40 [ 2520.069395] do_last+0x313/0xfc0 [ 2520.069400] ? nd_jump_root+0xe5/0x160 [ 2520.069404] ? path_init+0x437/0x520 [ 2520.069408] path_openat+0xf7/0x500 [ 2520.069413] ? do_raw_spin_unlock+0x75/0x190 [ 2520.069417] do_filp_open+0x99/0x140 [ 2520.069423] ? getname_flags+0x6e/0x330 [ 2520.069427] ? __check_object_size+0xff/0x256 [ 2520.069434] ? do_raw_spin_unlock+0x75/0x190 [ 2520.069437] ? _raw_spin_unlock+0x12/0x30 [ 2520.069441] do_sys_openat2+0x2b4/0x410 [ 2520.069446] do_sys_open+0x73/0xa0 [ 2520.069449] __x64_sys_openat+0x24/0x30 [ 2520.069452] do_syscall_64+0xc1/0x440 [ 2520.069458] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2520.069463] RIP: 0033:0x7fdc0afbf332 [ 2520.069469] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2520.069472] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2520.069477] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2520.069479] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2520.069481] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2520.069483] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2520.069485] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2520.410145] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2520.434820] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2520.438580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2520.442549] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2520.448730] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2520.455666] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2520.462933] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2520.467388] RAX: 00000000000062c4 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2520.470852] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2520.474997] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2520.478997] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2520.482381] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2520.485062] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2520.490229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2520.497104] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2520.501261] Call Trace: [ 2520.502246] [ 2520.503099] ? show_regs.cold.9+0x22/0x2f [ 2520.504785] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2520.506411] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2520.509227] ? watchdog+0x40/0x40 [ 2520.514253] ? hrtimer_interrupt+0x128/0x2c0 [ 2520.517892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2520.523426] ? apic_timer_interrupt+0xf/0x20 [ 2520.525178] [ 2520.526103] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2520.528204] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2520.531711] do_raw_spin_lock+0x177/0x190 [ 2520.533264] _raw_spin_lock+0x21/0x30 [ 2520.534803] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2520.537403] process_one_work+0x2c8/0x700 [ 2520.539017] worker_thread+0x296/0x6e0 [ 2520.540500] ? rescuer_thread+0x570/0x570 [ 2520.544672] kthread+0x1d1/0x200 [ 2520.546078] ? set_kthread_struct+0x70/0x70 [ 2520.549006] ret_from_fork+0x1f/0x30 [ 2536.054036] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2536.057599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2536.078809] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2536.084113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2536.088544] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2536.091886] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2536.102725] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2536.105843] RAX: 0000000000005a10 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2536.108637] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2536.111898] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2536.115171] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2536.118299] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2536.121569] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2536.125296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2536.127635] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2536.130992] Call Trace: [ 2536.132635] [ 2536.133745] ? show_regs.cold.9+0x22/0x2f [ 2536.135890] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2536.138734] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2536.140747] ? watchdog+0x40/0x40 [ 2536.142102] ? hrtimer_interrupt+0x128/0x2c0 [ 2536.143679] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2536.146131] ? apic_timer_interrupt+0xf/0x20 [ 2536.148270] [ 2536.149267] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2536.151718] do_raw_spin_lock+0x177/0x190 [ 2536.153351] _raw_spin_lock+0x21/0x30 [ 2536.154525] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2536.157179] ? do_raw_spin_unlock+0x75/0x190 [ 2536.159090] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2536.161346] ? do_wait_intr+0xf0/0xf0 [ 2536.162613] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2536.164318] mdc_close+0x53b/0xd70 [mdc] [ 2536.165975] lmv_close+0x18d/0x490 [lmv] [ 2536.167814] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2536.170626] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2536.172703] ll_md_close+0x274/0x800 [lustre] [ 2536.174579] ll_file_release+0x128/0xc50 [lustre] [ 2536.176389] __fput+0x10d/0x440 [ 2536.177526] ____fput+0x12/0x20 [ 2536.178376] task_work_run+0xc6/0x110 [ 2536.179542] exit_to_usermode_loop+0x1dd/0x1f0 [ 2536.180988] do_syscall_64+0x3d6/0x440 [ 2536.182564] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2536.185237] RIP: 0033:0x7fdc0afbfcd8 [ 2536.186703] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2536.195253] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2536.200681] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2536.204736] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2536.209070] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2536.212935] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2536.216487] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2544.048396] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 2544.057336] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2544.095252] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2544.104541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2544.111910] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2544.120021] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2544.140959] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2544.142702] RAX: 000000000000158c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2544.150183] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2544.159618] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2544.162549] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2544.169444] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2544.176190] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2544.181489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2544.185584] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2544.194828] Call Trace: [ 2544.196069] [ 2544.197076] ? show_regs.cold.9+0x22/0x2f [ 2544.199913] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2544.202143] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2544.204559] ? watchdog+0x40/0x40 [ 2544.206519] ? hrtimer_interrupt+0x128/0x2c0 [ 2544.208487] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2544.211591] ? apic_timer_interrupt+0xf/0x20 [ 2544.214470] [ 2544.215930] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2544.217849] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2544.224339] ? dump_stack+0xf6/0x10e [ 2544.225652] do_raw_spin_lock+0x177/0x190 [ 2544.235575] _raw_spin_lock+0x21/0x30 [ 2544.237149] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2544.241170] obd_mod_free+0x46/0x140 [ptlrpc] [ 2544.244037] mdc_commit_open+0x91/0xb0 [mdc] [ 2544.245802] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2544.250146] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2544.253514] after_reply+0xe2c/0x1780 [ptlrpc] [ 2544.256149] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2544.259008] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2544.261352] ? do_wait_intr+0xf0/0xf0 [ 2544.263230] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2544.266526] kthread+0x1d1/0x200 [ 2544.268525] ? set_kthread_struct+0x70/0x70 [ 2544.270631] ret_from_fork+0x1f/0x30 [ 2547.679509] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 522s! [ 2547.684871] Showing busy workqueues and worker pools: [ 2547.686908] workqueue events: flags=0x0 [ 2547.689069] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2547.691485] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2547.698591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2547.712382] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2547.719135] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2547.728704] pending: kfree_rcu_monitor [ 2547.733534] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2547.737554] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2547.754360] workqueue events_long: flags=0x0 [ 2547.769985] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2547.772276] pending: lu_objects_flush [obdclass] [ 2547.782114] workqueue events_unbound: flags=0x2 [ 2547.791598] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2547.796618] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2547.804839] workqueue events_power_efficient: flags=0x80 [ 2547.810765] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2547.818944] pending: neigh_periodic_work, check_lifetime [ 2547.822625] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2547.828539] pending: do_cache_clean, neigh_periodic_work [ 2547.831380] workqueue mm_percpu_wq: flags=0x8 [ 2547.835443] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2547.847603] pending: lru_add_drain_per_cpu [ 2547.850264] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2547.852931] pending: vmstat_update, lru_add_drain_per_cpu [ 2547.854703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2547.865623] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2547.870867] workqueue writeback: flags=0x4a [ 2547.873669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2547.877756] pending: wb_workfn, wb_workfn [ 2547.886119] workqueue xprtiod: flags=0xa [ 2547.891278] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2547.897771] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2547.900444] workqueue ipv6_addrconf: flags=0x40008 [ 2547.903606] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2547.909399] pending: addrconf_verify_work [ 2547.913177] workqueue knbd0-recv: flags=0x1a [ 2547.915846] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2547.919994] in-flight: 655:recv_work [ 2547.922114] workqueue ptlrpc_pinger: flags=0x2 [ 2547.924709] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2547.929955] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2547.933624] pool 8: cpus=0-3 flags=0x4 nice=0 hung=523s workers=4 idle: 36396 38 11188 [ 2547.938981] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2538s workers=2 idle: 658 [ 2548.051023] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 2548.054789] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 2548.055326] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2548.055402] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2548.055408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2548.055415] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2548.055722] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2548.055732] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2548.055736] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2548.055741] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2548.055743] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2548.055745] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2548.055747] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2548.055750] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2548.055753] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2548.055755] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2548.055757] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2548.055767] Call Trace: [ 2548.055772] [ 2548.055778] ? show_regs.cold.9+0x22/0x2f [ 2548.055788] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2548.055796] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2548.055800] ? watchdog+0x40/0x40 [ 2548.055804] ? hrtimer_interrupt+0x128/0x2c0 [ 2548.055808] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2548.055816] ? apic_timer_interrupt+0xf/0x20 [ 2548.055821] [ 2548.055824] ? native_safe_halt+0x2e/0x50 [ 2548.055829] kvm_wait+0xcd/0xe0 [ 2548.055976] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2548.055987] do_raw_spin_lock+0x177/0x190 [ 2548.055992] _raw_spin_lock+0x21/0x30 [ 2548.055996] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2548.056349] process_one_work+0x2c8/0x700 [ 2548.056357] worker_thread+0x296/0x6e0 [ 2548.056362] ? rescuer_thread+0x570/0x570 [ 2548.056366] kthread+0x1d1/0x200 [ 2548.056370] ? set_kthread_struct+0x70/0x70 [ 2548.056374] ret_from_fork+0x1f/0x30 [ 2548.420318] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2548.467149] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2548.480352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2548.492259] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2548.495085] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2548.508737] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2548.517491] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2548.519830] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2548.529633] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2548.535802] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2548.557475] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2548.563727] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2548.572519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2548.586489] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2548.596157] Call Trace: [ 2548.597034] [ 2548.600313] ? show_regs.cold.9+0x22/0x2f [ 2548.601818] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2548.605312] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2548.607655] ? watchdog+0x40/0x40 [ 2548.609980] ? hrtimer_interrupt+0x128/0x2c0 [ 2548.612595] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2548.619530] ? apic_timer_interrupt+0xf/0x20 [ 2548.624778] [ 2548.625541] ? native_safe_halt+0x2e/0x50 [ 2548.627764] kvm_wait+0xcd/0xe0 [ 2548.630514] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2548.636652] do_raw_spin_lock+0x177/0x190 [ 2548.642418] _raw_spin_lock+0x21/0x30 [ 2548.644698] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2548.648684] ? do_raw_spin_unlock+0x75/0x190 [ 2548.659996] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2548.668182] ? do_wait_intr+0xf0/0xf0 [ 2548.671568] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2548.676532] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2548.678873] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2548.682101] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2548.684067] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2548.692824] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2548.707369] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2548.708861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2548.713394] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2548.718512] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2548.721303] lmv_intent_open+0x338/0x1150 [lmv] [ 2548.724742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2548.730026] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2548.734792] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2548.738759] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2548.742935] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2548.749076] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2548.750648] ? do_raw_spin_unlock+0x75/0x190 [ 2548.752064] ? _raw_spin_unlock+0x12/0x30 [ 2548.761070] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2548.763862] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2548.766459] ll_atomic_open+0x36a/0xd70 [lustre] [ 2548.768940] lookup_open+0x356/0xb40 [ 2548.770150] do_last+0x313/0xfc0 [ 2548.771171] ? nd_jump_root+0xe5/0x160 [ 2548.778110] ? path_init+0x437/0x520 [ 2548.782083] path_openat+0xf7/0x500 [ 2548.785618] ? do_raw_spin_unlock+0x75/0x190 [ 2548.787642] do_filp_open+0x99/0x140 [ 2548.789464] ? getname_flags+0x6e/0x330 [ 2548.790746] ? __check_object_size+0xff/0x256 [ 2548.792958] ? do_raw_spin_unlock+0x75/0x190 [ 2548.794469] ? _raw_spin_unlock+0x12/0x30 [ 2548.797958] do_sys_openat2+0x2b4/0x410 [ 2548.799784] do_sys_open+0x73/0xa0 [ 2548.805780] __x64_sys_openat+0x24/0x30 [ 2548.810175] do_syscall_64+0xc1/0x440 [ 2548.812504] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2548.814207] RIP: 0033:0x7fdc0afbf332 [ 2548.817318] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2548.829523] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2548.833462] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2548.837729] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2548.840907] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2548.847708] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2548.855033] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2564.054194] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2564.058904] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2564.117078] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2564.130166] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2564.135411] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2564.146881] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2564.158951] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2564.182970] RAX: 0000000000001e89 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2564.197661] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2564.210281] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2564.215508] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2564.234650] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2564.246159] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2564.265875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2564.280123] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2564.292250] Call Trace: [ 2564.293040] [ 2564.293676] ? show_regs.cold.9+0x22/0x2f [ 2564.307109] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2564.312482] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2564.317517] ? watchdog+0x40/0x40 [ 2564.326542] ? hrtimer_interrupt+0x128/0x2c0 [ 2564.328009] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2564.340724] ? apic_timer_interrupt+0xf/0x20 [ 2564.346107] [ 2564.352024] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2564.358327] do_raw_spin_lock+0x177/0x190 [ 2564.364206] _raw_spin_lock+0x21/0x30 [ 2564.367764] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2564.370818] ? do_raw_spin_unlock+0x75/0x190 [ 2564.382913] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2564.393741] ? do_wait_intr+0xf0/0xf0 [ 2564.408196] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2564.411145] mdc_close+0x53b/0xd70 [mdc] [ 2564.412055] lmv_close+0x18d/0x490 [lmv] [ 2564.412943] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2564.418704] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2564.422749] ll_md_close+0x274/0x800 [lustre] [ 2564.426258] ll_file_release+0x128/0xc50 [lustre] [ 2564.429191] __fput+0x10d/0x440 [ 2564.431225] ____fput+0x12/0x20 [ 2564.433632] task_work_run+0xc6/0x110 [ 2564.436039] exit_to_usermode_loop+0x1dd/0x1f0 [ 2564.439055] do_syscall_64+0x3d6/0x440 [ 2564.441246] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2564.444023] RIP: 0033:0x7fdc0afbfcd8 [ 2564.445746] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2564.454619] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2564.457662] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2564.461251] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2564.464730] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2564.468145] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2564.471288] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2572.051657] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2572.058249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2572.101102] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2572.114080] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2572.130554] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2572.142450] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2572.170965] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2572.182283] RAX: 0000000000007317 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2572.198877] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2572.206753] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2572.215267] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2572.220718] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2572.233706] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2572.241115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2572.245345] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2572.259047] Call Trace: [ 2572.262385] [ 2572.268592] ? show_regs.cold.9+0x22/0x2f [ 2572.270114] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2572.277098] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2572.282370] ? watchdog+0x40/0x40 [ 2572.284212] ? hrtimer_interrupt+0x128/0x2c0 [ 2572.295305] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2572.298142] ? apic_timer_interrupt+0xf/0x20 [ 2572.299707] [ 2572.305930] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2572.316035] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2572.320323] ? dump_stack+0xf6/0x10e [ 2572.322704] do_raw_spin_lock+0x177/0x190 [ 2572.325237] _raw_spin_lock+0x21/0x30 [ 2572.327484] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2572.333984] obd_mod_free+0x46/0x140 [ptlrpc] [ 2572.336125] mdc_commit_open+0x91/0xb0 [mdc] [ 2572.337224] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2572.342452] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2572.348002] after_reply+0xe2c/0x1780 [ptlrpc] [ 2572.355352] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2572.361126] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2572.372938] ? do_wait_intr+0xf0/0xf0 [ 2572.374451] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2572.395763] kthread+0x1d1/0x200 [ 2572.396857] ? set_kthread_struct+0x70/0x70 [ 2572.401275] ret_from_fork+0x1f/0x30 [ 2576.051052] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 2576.056766] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 2576.057721] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2576.057822] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2576.057831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2576.057840] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2576.059088] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2576.059106] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2576.059111] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2576.059116] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2576.059119] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2576.059121] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2576.059123] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2576.059126] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2576.059129] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2576.059132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2576.059135] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2576.059144] Call Trace: [ 2576.059152] [ 2576.059160] ? show_regs.cold.9+0x22/0x2f [ 2576.059175] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2576.059183] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2576.059187] ? watchdog+0x40/0x40 [ 2576.059192] ? hrtimer_interrupt+0x128/0x2c0 [ 2576.059196] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2576.059205] ? apic_timer_interrupt+0xf/0x20 [ 2576.059209] [ 2576.059212] ? native_safe_halt+0x2e/0x50 [ 2576.059217] kvm_wait+0xcd/0xe0 [ 2576.059239] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2576.059252] do_raw_spin_lock+0x177/0x190 [ 2576.059257] _raw_spin_lock+0x21/0x30 [ 2576.059262] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2576.060143] process_one_work+0x2c8/0x700 [ 2576.060153] worker_thread+0x296/0x6e0 [ 2576.060160] ? rescuer_thread+0x570/0x570 [ 2576.060164] kthread+0x1d1/0x200 [ 2576.060169] ? set_kthread_struct+0x70/0x70 [ 2576.060172] ret_from_fork+0x1f/0x30 [ 2576.330901] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2576.383551] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2576.401979] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2576.414601] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2576.423562] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2576.442236] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2576.451830] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2576.458148] RDX: 00000000000060fc RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2576.463198] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2576.473194] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2576.481991] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2576.489532] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2576.500121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2576.505846] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2576.513084] Call Trace: [ 2576.518843] [ 2576.519618] ? show_regs.cold.9+0x22/0x2f [ 2576.521026] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2576.527717] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2576.539715] ? watchdog+0x40/0x40 [ 2576.542717] ? hrtimer_interrupt+0x128/0x2c0 [ 2576.546787] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2576.550805] ? apic_timer_interrupt+0xf/0x20 [ 2576.553917] [ 2576.556776] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2576.560758] do_raw_spin_lock+0x177/0x190 [ 2576.564104] _raw_spin_lock+0x21/0x30 [ 2576.567961] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2576.581072] ? do_raw_spin_unlock+0x75/0x190 [ 2576.583770] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2576.594267] ? do_wait_intr+0xf0/0xf0 [ 2576.599150] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2576.607240] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2576.617781] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2576.624928] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2576.633664] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2576.640670] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2576.643572] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2576.649494] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2576.664218] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2576.672729] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2576.682772] lmv_intent_open+0x338/0x1150 [lmv] [ 2576.684688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2576.687153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2576.695372] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2576.702807] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2576.709299] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2576.722773] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2576.728857] ? do_raw_spin_unlock+0x75/0x190 [ 2576.733631] ? _raw_spin_unlock+0x12/0x30 [ 2576.737531] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2576.743854] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2576.759297] ll_atomic_open+0x36a/0xd70 [lustre] [ 2576.764061] lookup_open+0x356/0xb40 [ 2576.769959] do_last+0x313/0xfc0 [ 2576.771105] ? nd_jump_root+0xe5/0x160 [ 2576.774396] ? path_init+0x437/0x520 [ 2576.775929] path_openat+0xf7/0x500 [ 2576.777840] ? do_raw_spin_unlock+0x75/0x190 [ 2576.780331] do_filp_open+0x99/0x140 [ 2576.782160] ? getname_flags+0x6e/0x330 [ 2576.784204] ? __check_object_size+0xff/0x256 [ 2576.797199] ? do_raw_spin_unlock+0x75/0x190 [ 2576.799812] ? _raw_spin_unlock+0x12/0x30 [ 2576.805924] do_sys_openat2+0x2b4/0x410 [ 2576.808012] do_sys_open+0x73/0xa0 [ 2576.812697] __x64_sys_openat+0x24/0x30 [ 2576.814722] do_syscall_64+0xc1/0x440 [ 2576.823927] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2576.825673] RIP: 0033:0x7fdc0afbf332 [ 2576.831412] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2576.850481] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2576.859165] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2576.875246] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2576.886093] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2576.898202] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2576.904191] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2578.399087] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 553s! [ 2578.404574] Showing busy workqueues and worker pools: [ 2578.407487] workqueue events: flags=0x0 [ 2578.409627] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2578.412409] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2578.415573] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2578.418799] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2578.423415] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2578.426482] pending: kfree_rcu_monitor [ 2578.428628] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2578.431668] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2578.436582] workqueue events_long: flags=0x0 [ 2578.438763] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2578.441459] pending: lu_objects_flush [obdclass] [ 2578.444743] workqueue events_unbound: flags=0x2 [ 2578.447051] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2578.449419] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2578.454599] workqueue events_power_efficient: flags=0x80 [ 2578.456715] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2578.459834] pending: neigh_periodic_work, check_lifetime [ 2578.462996] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2578.466263] pending: do_cache_clean, neigh_periodic_work [ 2578.468532] workqueue mm_percpu_wq: flags=0x8 [ 2578.470454] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2578.474643] pending: lru_add_drain_per_cpu [ 2578.477217] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2578.480383] pending: vmstat_update, lru_add_drain_per_cpu [ 2578.482948] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2578.485421] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2578.488339] workqueue writeback: flags=0x4a [ 2578.490282] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2578.499194] pending: wb_workfn, wb_workfn [ 2578.502044] workqueue xprtiod: flags=0xa [ 2578.504735] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2578.509258] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2578.514186] workqueue ipv6_addrconf: flags=0x40008 [ 2578.517460] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2578.521153] pending: addrconf_verify_work [ 2578.523777] workqueue knbd0-recv: flags=0x1a [ 2578.526678] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2578.530173] in-flight: 655:recv_work [ 2578.532436] workqueue ptlrpc_pinger: flags=0x2 [ 2578.535113] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2578.537041] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2578.542464] pool 8: cpus=0-3 flags=0x4 nice=0 hung=553s workers=4 idle: 36396 38 11188 [ 2578.555441] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2569s workers=2 idle: 658 [ 2592.055976] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 2592.060404] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2592.114717] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2592.131716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2592.137997] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2592.146961] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2592.162677] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2592.172373] RAX: 0000000000002d1b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2592.186996] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2592.189970] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2592.196377] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2592.203642] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2592.211830] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2592.218452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2592.220369] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2592.228521] Call Trace: [ 2592.229372] [ 2592.230111] ? show_regs.cold.9+0x22/0x2f [ 2592.238780] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2592.240441] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2592.247777] ? watchdog+0x40/0x40 [ 2592.251398] ? hrtimer_interrupt+0x128/0x2c0 [ 2592.256587] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2592.260780] ? apic_timer_interrupt+0xf/0x20 [ 2592.264398] [ 2592.265438] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2592.267617] do_raw_spin_lock+0x177/0x190 [ 2592.272339] _raw_spin_lock+0x21/0x30 [ 2592.277695] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2592.282654] ? do_raw_spin_unlock+0x75/0x190 [ 2592.284792] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2592.287488] ? do_wait_intr+0xf0/0xf0 [ 2592.291968] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2592.298320] mdc_close+0x53b/0xd70 [mdc] [ 2592.300834] lmv_close+0x18d/0x490 [lmv] [ 2592.303004] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2592.306176] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2592.307961] ll_md_close+0x274/0x800 [lustre] [ 2592.315496] ll_file_release+0x128/0xc50 [lustre] [ 2592.320267] __fput+0x10d/0x440 [ 2592.321848] ____fput+0x12/0x20 [ 2592.323649] task_work_run+0xc6/0x110 [ 2592.325444] exit_to_usermode_loop+0x1dd/0x1f0 [ 2592.328301] do_syscall_64+0x3d6/0x440 [ 2592.330081] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2592.331735] RIP: 0033:0x7fdc0afbfcd8 [ 2592.338458] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2592.353450] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2592.359005] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2592.363118] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2592.371153] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2592.375831] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2592.379349] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2604.048712] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 2604.051024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 2604.055661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2604.055740] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2604.055748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2604.055751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2604.055765] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2604.055769] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2604.055774] RAX: 0000000000007108 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2604.055777] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2604.055779] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2604.055781] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2604.055783] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2604.055787] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2604.055789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2604.055792] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2604.055801] Call Trace: [ 2604.055806] [ 2604.055813] ? show_regs.cold.9+0x22/0x2f [ 2604.055824] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2604.055830] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2604.055835] ? watchdog+0x40/0x40 [ 2604.055839] ? hrtimer_interrupt+0x128/0x2c0 [ 2604.055843] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2604.055853] ? apic_timer_interrupt+0xf/0x20 [ 2604.055858] [ 2604.055861] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2604.055865] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2604.055951] ? dump_stack+0xf6/0x10e [ 2604.055957] do_raw_spin_lock+0x177/0x190 [ 2604.055962] _raw_spin_lock+0x21/0x30 [ 2604.055968] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2604.058156] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 2604.058164] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2604.059127] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2604.059133] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2604.059136] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2604.059147] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2604.059151] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2604.059155] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2604.059157] RDX: 0000000000002060 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2604.059160] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2604.059162] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2604.059164] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2604.059166] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2604.059170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2604.059172] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2604.060015] Call Trace: [ 2604.060028] [ 2604.060034] ? show_regs.cold.9+0x22/0x2f [ 2604.060046] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2604.060052] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2604.060056] ? watchdog+0x40/0x40 [ 2604.060060] ? hrtimer_interrupt+0x128/0x2c0 [ 2604.060063] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2604.060071] ? apic_timer_interrupt+0xf/0x20 [ 2604.060075] [ 2604.060078] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2604.060084] do_raw_spin_lock+0x177/0x190 [ 2604.060087] _raw_spin_lock+0x21/0x30 [ 2604.060093] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2604.060451] ? do_raw_spin_unlock+0x75/0x190 [ 2604.060456] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2604.060655] ? do_wait_intr+0xf0/0xf0 [ 2604.060663] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2604.060862] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2604.061146] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2604.061449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2604.061563] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2604.061601] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2604.061634] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2604.061669] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2604.062113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2604.062177] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2604.062183] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2604.062189] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2604.063462] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2604.063478] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2604.063482] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2604.063486] RAX: 0000000000001e02 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2604.063488] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2604.063490] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2604.063492] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2604.063495] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2604.063498] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2604.063501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2604.063503] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2604.063511] Call Trace: [ 2604.063517] [ 2604.063523] ? show_regs.cold.9+0x22/0x2f [ 2604.063534] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2604.063540] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2604.063545] ? watchdog+0x40/0x40 [ 2604.063549] ? hrtimer_interrupt+0x128/0x2c0 [ 2604.061758] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2604.063554] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2604.063565] ? apic_timer_interrupt+0xf/0x20 [ 2604.063570] [ 2604.063572] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2604.063577] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2604.063582] do_raw_spin_lock+0x177/0x190 [ 2604.063587] _raw_spin_lock+0x21/0x30 [ 2604.063774] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2604.063815] lmv_intent_open+0x338/0x1150 [lmv] [ 2604.063593] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2604.064206] process_one_work+0x2c8/0x700 [ 2604.064214] worker_thread+0x296/0x6e0 [ 2604.064217] ? rescuer_thread+0x570/0x570 [ 2604.064220] kthread+0x1d1/0x200 [ 2604.064225] ? set_kthread_struct+0x70/0x70 [ 2604.064229] ret_from_fork+0x1f/0x30 [ 2604.063853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2604.109812] obd_mod_free+0x46/0x140 [ptlrpc] [ 2604.111772] mdc_commit_open+0x91/0xb0 [mdc] [ 2604.122966] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2604.123092] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2604.123127] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2604.123211] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2604.123309] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2604.123388] ? do_raw_spin_unlock+0x75/0x190 [ 2604.123397] ? _raw_spin_unlock+0x12/0x30 [ 2604.123404] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2604.123430] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2604.123448] ll_atomic_open+0x36a/0xd70 [lustre] [ 2604.123530] lookup_open+0x356/0xb40 [ 2604.123542] do_last+0x313/0xfc0 [ 2604.123547] ? nd_jump_root+0xe5/0x160 [ 2604.123551] ? path_init+0x437/0x520 [ 2604.123555] path_openat+0xf7/0x500 [ 2604.123559] ? do_raw_spin_unlock+0x75/0x190 [ 2604.123563] do_filp_open+0x99/0x140 [ 2604.123569] ? getname_flags+0x6e/0x330 [ 2604.123573] ? __check_object_size+0xff/0x256 [ 2604.123579] ? do_raw_spin_unlock+0x75/0x190 [ 2604.123583] ? _raw_spin_unlock+0x12/0x30 [ 2604.123587] do_sys_openat2+0x2b4/0x410 [ 2604.123591] do_sys_open+0x73/0xa0 [ 2604.123595] __x64_sys_openat+0x24/0x30 [ 2604.123598] do_syscall_64+0xc1/0x440 [ 2604.123604] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2604.123611] RIP: 0033:0x7fdc0afbf332 [ 2604.123618] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2604.123622] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2604.123628] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2604.123630] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2604.123632] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2604.123634] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2604.123637] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2604.591019] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2604.598643] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2604.607756] after_reply+0xe2c/0x1780 [ptlrpc] [ 2604.611835] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2604.620632] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2604.627243] ? do_wait_intr+0xf0/0xf0 [ 2604.634643] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2604.639373] kthread+0x1d1/0x200 [ 2604.641947] ? set_kthread_struct+0x70/0x70 [ 2604.643773] ret_from_fork+0x1f/0x30 [ 2608.607280] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 583s! [ 2608.610103] Showing busy workqueues and worker pools: [ 2608.618406] workqueue events: flags=0x0 [ 2608.619820] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2608.637251] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2608.640254] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2608.646115] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2608.657469] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2608.661385] pending: kfree_rcu_monitor [ 2608.662917] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2608.667298] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2608.676561] workqueue events_long: flags=0x0 [ 2608.680164] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2608.691588] pending: lu_objects_flush [obdclass] [ 2608.695217] workqueue events_unbound: flags=0x2 [ 2608.702195] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2608.706119] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2608.711708] workqueue events_power_efficient: flags=0x80 [ 2608.718038] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2608.722662] pending: neigh_periodic_work, check_lifetime [ 2608.726679] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2608.732763] pending: do_cache_clean, neigh_periodic_work [ 2608.734712] workqueue mm_percpu_wq: flags=0x8 [ 2608.743247] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2608.747182] pending: lru_add_drain_per_cpu [ 2608.750156] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2608.753306] pending: vmstat_update, lru_add_drain_per_cpu [ 2608.767545] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2608.773890] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2608.778719] workqueue writeback: flags=0x4a [ 2608.789922] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2608.792163] pending: wb_workfn, wb_workfn [ 2608.800572] workqueue xprtiod: flags=0xa [ 2608.804326] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2608.808385] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2608.814931] workqueue ipv6_addrconf: flags=0x40008 [ 2608.817473] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2608.820645] pending: addrconf_verify_work [ 2608.828123] workqueue knbd0-recv: flags=0x1a [ 2608.829635] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2608.834658] in-flight: 655:recv_work [ 2608.837208] workqueue ptlrpc_pinger: flags=0x2 [ 2608.839416] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2608.842552] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2608.846368] pool 8: cpus=0-3 flags=0x4 nice=0 hung=583s workers=4 idle: 36396 38 11188 [ 2608.855620] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2599s workers=2 idle: 658 [ 2619.357149] rcu: INFO: rcu_sched self-detected stall on CPU [ 2619.363114] rcu: 0-...!: (340473 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 2619.388212] (t=600014 jiffies g=270761 q=2837) [ 2619.400481] rcu: rcu_sched kthread starved for 594946 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 2619.414384] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2619.424475] rcu: RCU grace-period kthread stack dump: [ 2619.433961] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2619.451548] Call Trace: [ 2619.452433] __schedule+0x351/0xcb0 [ 2619.461789] schedule+0xc0/0x180 [ 2619.464145] schedule_timeout+0xb4/0x190 [ 2619.473798] ? do_raw_spin_unlock+0x75/0x190 [ 2619.485011] ? __next_timer_interrupt+0x160/0x160 [ 2619.490063] ? prepare_to_swait_event+0xf3/0x140 [ 2619.493160] rcu_gp_kthread+0x8dd/0x15b0 [ 2619.495312] ? force_qs_rnp+0x2c0/0x2c0 [ 2619.496731] kthread+0x1d1/0x200 [ 2619.498241] ? set_kthread_struct+0x70/0x70 [ 2619.508145] ret_from_fork+0x1f/0x30 [ 2619.512160] rcu: Stack dump where RCU GP kthread last ran: [ 2619.521785] Sending NMI from CPU 0 to CPUs 3: [ 2619.529793] NMI backtrace for cpu 3 [ 2619.529802] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2619.529804] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2619.529805] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2619.529807] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2619.529810] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 [ 2619.529813] RAX: 0000000000006b0c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2619.529814] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2619.529816] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2619.529818] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2619.529819] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2619.529820] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2619.529821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2619.529822] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2619.529823] Call Trace: [ 2619.529824] [ 2619.529825] ? show_regs.cold.9+0x22/0x2f [ 2619.529827] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2619.529828] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2619.529829] ? nmi_handle+0xbd/0x210 [ 2619.529830] ? default_do_nmi+0x4b/0x1b0 [ 2619.529831] ? do_nmi+0x280/0x380 [ 2619.529832] ? end_repeat_nmi+0x16/0x1f [ 2619.529833] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2619.529834] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2619.529835] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2619.529836] [ 2619.529837] do_raw_spin_lock+0x177/0x190 [ 2619.529838] _raw_spin_lock+0x21/0x30 [ 2619.529839] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2619.529840] ? do_raw_spin_unlock+0x75/0x190 [ 2619.529841] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2619.529842] ? do_wait_intr+0xf0/0xf0 [ 2619.529843] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2619.529844] mdc_close+0x53b/0xd70 [mdc] [ 2619.529845] lmv_close+0x18d/0x490 [lmv] [ 2619.529846] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2619.529847] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2619.529848] ll_md_close+0x274/0x800 [lustre] [ 2619.529849] ll_file_release+0x128/0xc50 [lustre] [ 2619.529850] __fput+0x10d/0x440 [ 2619.529851] ____fput+0x12/0x20 [ 2619.529852] task_work_run+0xc6/0x110 [ 2619.529853] exit_to_usermode_loop+0x1dd/0x1f0 [ 2619.529854] do_syscall_64+0x3d6/0x440 [ 2619.529855] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2619.529856] RIP: 0033:0x7fdc0afbfcd8 [ 2619.529858] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2619.529860] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2619.529862] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2619.529864] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2619.529947] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2619.529949] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2619.529950] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2619.531306] NMI backtrace for cpu 0 [ 2619.869819] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2619.882966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2619.890309] Call Trace: [ 2619.896114] [ 2619.897231] ? dump_stack+0xbb/0x10e [ 2619.903314] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2619.906693] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2619.911163] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2619.913043] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2619.932023] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2619.940379] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2619.941949] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2619.952342] ? update_process_times+0x6d/0xc0 [ 2619.960367] ? tick_sched_handle.isra.18+0x27/0x90 [ 2619.969859] ? tick_sched_timer+0x53/0xb0 [ 2619.981519] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2619.983132] ? tick_sched_handle.isra.18+0x90/0x90 [ 2619.991379] ? hrtimer_interrupt+0x128/0x2c0 [ 2619.994962] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2620.007065] ? apic_timer_interrupt+0xf/0x20 [ 2620.020433] [ 2620.024957] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2620.037014] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2620.041069] ? dump_stack+0xf6/0x10e [ 2620.047686] ? do_raw_spin_lock+0x177/0x190 [ 2620.049256] ? _raw_spin_lock+0x21/0x30 [ 2620.065060] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2620.083397] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2620.085434] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2620.091935] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2620.105387] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2620.115906] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2620.125224] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2620.132927] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2620.143329] ? do_wait_intr+0xf0/0xf0 [ 2620.144746] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2620.156704] ? kthread+0x1d1/0x200 [ 2620.159991] ? set_kthread_struct+0x70/0x70 [ 2620.171110] ? ret_from_fork+0x1f/0x30 [ 2632.051182] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 2632.053354] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 2632.065861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2632.066047] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2632.066055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2632.066060] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2632.066691] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2632.066827] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2632.066833] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2632.066838] RAX: 0000000000003d6d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2632.066840] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2632.066843] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2632.066845] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2632.066847] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2632.066855] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2632.066859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2632.066861] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2632.066869] Call Trace: [ 2632.066875] [ 2632.066881] ? show_regs.cold.9+0x22/0x2f [ 2632.066896] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2632.066903] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2632.066907] ? watchdog+0x40/0x40 [ 2632.066912] ? hrtimer_interrupt+0x128/0x2c0 [ 2632.066916] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2632.066931] ? apic_timer_interrupt+0xf/0x20 [ 2632.066935] [ 2632.066938] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2632.066944] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2632.066953] do_raw_spin_lock+0x177/0x190 [ 2632.066958] _raw_spin_lock+0x21/0x30 [ 2632.066963] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2632.135724] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2632.135817] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2632.135824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2632.135827] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2632.135843] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2632.135847] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2632.135852] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2632.135968] RDX: 00000000000057c2 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2632.135972] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2632.135975] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2632.135977] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2632.135981] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2632.135984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2632.135987] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2632.135994] Call Trace: [ 2632.136000] [ 2632.136007] ? show_regs.cold.9+0x22/0x2f [ 2632.136019] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2632.136028] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2632.136033] ? watchdog+0x40/0x40 [ 2632.136038] ? hrtimer_interrupt+0x128/0x2c0 [ 2632.136043] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2632.136054] ? apic_timer_interrupt+0xf/0x20 [ 2632.136059] [ 2632.136062] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2632.136142] do_raw_spin_lock+0x177/0x190 [ 2632.136150] _raw_spin_lock+0x21/0x30 [ 2632.136156] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2632.137132] ? do_raw_spin_unlock+0x75/0x190 [ 2632.137138] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2632.137336] ? do_wait_intr+0xf0/0xf0 [ 2632.137345] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2632.138622] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2632.139129] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2632.140342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2632.150162] process_one_work+0x2c8/0x700 [ 2632.150182] worker_thread+0x296/0x6e0 [ 2632.150187] ? rescuer_thread+0x570/0x570 [ 2632.150192] kthread+0x1d1/0x200 [ 2632.150196] ? set_kthread_struct+0x70/0x70 [ 2632.150200] ret_from_fork+0x1f/0x30 [ 2632.516740] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2632.525672] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2632.530010] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2632.539627] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2632.543456] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2632.550032] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2632.551964] lmv_intent_open+0x338/0x1150 [lmv] [ 2632.557342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2632.561419] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2632.568652] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2632.572253] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2632.577719] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2632.581696] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2632.584619] ? do_raw_spin_unlock+0x75/0x190 [ 2632.586065] ? _raw_spin_unlock+0x12/0x30 [ 2632.594138] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2632.596686] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2632.601312] ll_atomic_open+0x36a/0xd70 [lustre] [ 2632.603859] lookup_open+0x356/0xb40 [ 2632.605484] do_last+0x313/0xfc0 [ 2632.607244] ? nd_jump_root+0xe5/0x160 [ 2632.609946] ? path_init+0x437/0x520 [ 2632.612972] path_openat+0xf7/0x500 [ 2632.615298] ? do_raw_spin_unlock+0x75/0x190 [ 2632.617179] do_filp_open+0x99/0x140 [ 2632.618740] ? getname_flags+0x6e/0x330 [ 2632.620651] ? __check_object_size+0xff/0x256 [ 2632.622739] ? do_raw_spin_unlock+0x75/0x190 [ 2632.624083] ? _raw_spin_unlock+0x12/0x30 [ 2632.625394] do_sys_openat2+0x2b4/0x410 [ 2632.630508] do_sys_open+0x73/0xa0 [ 2632.632161] __x64_sys_openat+0x24/0x30 [ 2632.633967] do_syscall_64+0xc1/0x440 [ 2632.635472] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2632.644468] RIP: 0033:0x7fdc0afbf332 [ 2632.647210] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2632.662463] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2632.667615] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2632.670453] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2632.675061] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2632.685595] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2632.687382] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2639.327950] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 614s! [ 2639.333404] Showing busy workqueues and worker pools: [ 2639.336880] workqueue events: flags=0x0 [ 2639.339674] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2639.344363] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2639.349872] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2639.353876] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2639.360492] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2639.368331] pending: kfree_rcu_monitor [ 2639.371318] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2639.374984] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2639.379556] workqueue events_long: flags=0x0 [ 2639.382927] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2639.390483] pending: lu_objects_flush [obdclass] [ 2639.393806] workqueue events_unbound: flags=0x2 [ 2639.397621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2639.399474] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2639.407379] workqueue events_power_efficient: flags=0x80 [ 2639.412749] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2639.418517] pending: neigh_periodic_work, check_lifetime [ 2639.427634] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2639.437627] pending: do_cache_clean, neigh_periodic_work [ 2639.439565] workqueue mm_percpu_wq: flags=0x8 [ 2639.442376] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2639.448182] pending: lru_add_drain_per_cpu [ 2639.454215] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2639.462297] pending: vmstat_update, lru_add_drain_per_cpu [ 2639.467955] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2639.472157] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2639.476292] workqueue writeback: flags=0x4a [ 2639.478958] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2639.485711] pending: wb_workfn, wb_workfn [ 2639.487667] workqueue xprtiod: flags=0xa [ 2639.489717] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2639.496038] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2639.501669] workqueue ipv6_addrconf: flags=0x40008 [ 2639.504422] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2639.507401] pending: addrconf_verify_work [ 2639.510823] workqueue knbd0-recv: flags=0x1a [ 2639.514512] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2639.520741] in-flight: 655:recv_work [ 2639.533523] workqueue ptlrpc_pinger: flags=0x2 [ 2639.534929] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2639.546412] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2639.552280] pool 8: cpus=0-3 flags=0x4 nice=0 hung=614s workers=4 idle: 36396 38 11188 [ 2639.557665] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2630s workers=2 idle: 658 [ 2644.067101] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2644.070152] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2644.123943] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2644.135689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2644.144062] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2644.148758] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2644.161790] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2644.167791] RAX: 0000000000007881 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2644.173766] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2644.180298] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2644.185166] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2644.189776] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2644.197353] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2644.203315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2644.207019] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2644.211586] Call Trace: [ 2644.214864] [ 2644.216098] ? show_regs.cold.9+0x22/0x2f [ 2644.220106] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2644.224377] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2644.226856] ? watchdog+0x40/0x40 [ 2644.229397] ? hrtimer_interrupt+0x128/0x2c0 [ 2644.231367] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2644.234062] ? apic_timer_interrupt+0xf/0x20 [ 2644.238878] [ 2644.249988] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2644.252938] do_raw_spin_lock+0x177/0x190 [ 2644.258503] _raw_spin_lock+0x21/0x30 [ 2644.261076] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2644.263122] ? do_raw_spin_unlock+0x75/0x190 [ 2644.273894] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2644.278670] ? do_wait_intr+0xf0/0xf0 [ 2644.280143] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2644.284136] mdc_close+0x53b/0xd70 [mdc] [ 2644.289411] lmv_close+0x18d/0x490 [lmv] [ 2644.294428] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2644.303575] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2644.307118] ll_md_close+0x274/0x800 [lustre] [ 2644.312937] ll_file_release+0x128/0xc50 [lustre] [ 2644.323760] __fput+0x10d/0x440 [ 2644.325723] ____fput+0x12/0x20 [ 2644.329829] task_work_run+0xc6/0x110 [ 2644.338499] exit_to_usermode_loop+0x1dd/0x1f0 [ 2644.350683] do_syscall_64+0x3d6/0x440 [ 2644.354591] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2644.357117] RIP: 0033:0x7fdc0afbfcd8 [ 2644.359703] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2644.370841] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2644.381041] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2644.390885] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2644.397080] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2644.403634] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2644.409959] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2660.055080] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2660.057523] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2660.069635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2660.069710] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2660.069718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2660.069724] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2660.070756] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2660.070769] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2660.070773] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2660.070778] RAX: 0000000000005e8c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2660.070780] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2660.070782] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2660.070784] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2660.070787] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2660.070790] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2660.070792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2660.070794] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2660.070803] Call Trace: [ 2660.070808] [ 2660.070814] ? show_regs.cold.9+0x22/0x2f [ 2660.070824] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2660.070831] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2660.070835] ? watchdog+0x40/0x40 [ 2660.070839] ? hrtimer_interrupt+0x128/0x2c0 [ 2660.070844] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2660.070851] ? apic_timer_interrupt+0xf/0x20 [ 2660.070855] [ 2660.070858] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2660.070862] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2660.074174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2660.074927] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2660.074935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2660.074937] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2660.074952] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2660.074957] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2660.074961] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2660.074964] RDX: 0000000000004066 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2660.074966] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2660.074969] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2660.074971] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2660.074974] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2660.074978] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2660.074980] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2660.074988] Call Trace: [ 2660.074993] [ 2660.075000] ? show_regs.cold.9+0x22/0x2f [ 2660.075011] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2660.075018] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2660.075022] ? watchdog+0x40/0x40 [ 2660.075027] ? hrtimer_interrupt+0x128/0x2c0 [ 2660.075030] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2660.075040] ? apic_timer_interrupt+0xf/0x20 [ 2660.075045] [ 2660.075048] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2660.075053] do_raw_spin_lock+0x177/0x190 [ 2660.075057] _raw_spin_lock+0x21/0x30 [ 2660.075063] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2660.075650] ? do_raw_spin_unlock+0x75/0x190 [ 2660.075657] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2660.075858] ? do_wait_intr+0xf0/0xf0 [ 2660.075928] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2660.076126] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2660.076432] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2660.076716] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2660.076829] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2660.076933] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2660.076967] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2660.076995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2660.077079] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2660.077389] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2660.077417] lmv_intent_open+0x338/0x1150 [lmv] [ 2660.077448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2660.077600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2660.077686] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2660.077709] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2660.077796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2660.077936] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2660.078020] ? do_raw_spin_unlock+0x75/0x190 [ 2660.078026] ? _raw_spin_unlock+0x12/0x30 [ 2660.078033] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2660.078060] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2660.078080] ll_atomic_open+0x36a/0xd70 [lustre] [ 2660.078165] lookup_open+0x356/0xb40 [ 2660.078175] do_last+0x313/0xfc0 [ 2660.078180] ? nd_jump_root+0xe5/0x160 [ 2660.078185] ? path_init+0x437/0x520 [ 2660.078189] path_openat+0xf7/0x500 [ 2660.078194] ? do_raw_spin_unlock+0x75/0x190 [ 2660.078198] do_filp_open+0x99/0x140 [ 2660.078204] ? getname_flags+0x6e/0x330 [ 2660.078209] ? __check_object_size+0xff/0x256 [ 2660.078217] ? do_raw_spin_unlock+0x75/0x190 [ 2660.078220] ? _raw_spin_unlock+0x12/0x30 [ 2660.078225] do_sys_openat2+0x2b4/0x410 [ 2660.078230] do_sys_open+0x73/0xa0 [ 2660.078233] __x64_sys_openat+0x24/0x30 [ 2660.078565] do_syscall_64+0xc1/0x440 [ 2660.078575] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2660.078582] RIP: 0033:0x7fdc0afbf332 [ 2660.078590] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2660.078594] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2660.078599] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2660.078601] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2660.078604] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2660.078606] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2660.078608] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2660.469083] do_raw_spin_lock+0x177/0x190 [ 2660.470854] _raw_spin_lock+0x21/0x30 [ 2660.474169] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2660.478064] process_one_work+0x2c8/0x700 [ 2660.482786] worker_thread+0x296/0x6e0 [ 2660.486758] ? rescuer_thread+0x570/0x570 [ 2660.488849] kthread+0x1d1/0x200 [ 2660.489880] ? set_kthread_struct+0x70/0x70 [ 2660.494853] ret_from_fork+0x1f/0x30 [ 2664.052147] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 2664.061381] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2664.108758] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2664.123119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2664.130530] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2664.137697] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2664.165744] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2664.172757] RAX: 0000000000006369 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2664.181013] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2664.187517] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2664.204553] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2664.214828] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2664.220029] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2664.231946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2664.240902] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2664.254371] Call Trace: [ 2664.255353] [ 2664.256125] ? show_regs.cold.9+0x22/0x2f [ 2664.265534] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2664.267194] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2664.271781] ? watchdog+0x40/0x40 [ 2664.272969] ? hrtimer_interrupt+0x128/0x2c0 [ 2664.274156] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2664.279017] ? apic_timer_interrupt+0xf/0x20 [ 2664.282527] [ 2664.284008] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2664.291581] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2664.298693] ? dump_stack+0xf6/0x10e [ 2664.300508] do_raw_spin_lock+0x177/0x190 [ 2664.308784] _raw_spin_lock+0x21/0x30 [ 2664.313649] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2664.324213] obd_mod_free+0x46/0x140 [ptlrpc] [ 2664.334893] mdc_commit_open+0x91/0xb0 [mdc] [ 2664.336475] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2664.348573] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2664.354021] after_reply+0xe2c/0x1780 [ptlrpc] [ 2664.357274] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2664.366226] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2664.367981] ? do_wait_intr+0xf0/0xf0 [ 2664.376744] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2664.384557] kthread+0x1d1/0x200 [ 2664.386719] ? set_kthread_struct+0x70/0x70 [ 2664.391544] ret_from_fork+0x1f/0x30 [ 2670.047041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 645s! [ 2670.052772] Showing busy workqueues and worker pools: [ 2670.057663] workqueue events: flags=0x0 [ 2670.059515] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2670.062556] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2670.065821] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2670.068944] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2670.073141] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2670.075663] pending: kfree_rcu_monitor [ 2670.077498] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2670.080536] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2670.084648] workqueue events_long: flags=0x0 [ 2670.085861] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2670.088491] pending: lu_objects_flush [obdclass] [ 2670.090622] workqueue events_unbound: flags=0x2 [ 2670.092355] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2670.094870] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2670.099187] workqueue events_power_efficient: flags=0x80 [ 2670.101108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2670.104015] pending: neigh_periodic_work, check_lifetime [ 2670.106105] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2670.108491] pending: do_cache_clean, neigh_periodic_work [ 2670.110987] workqueue mm_percpu_wq: flags=0x8 [ 2670.112910] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2670.115421] pending: lru_add_drain_per_cpu [ 2670.117071] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2670.119626] pending: vmstat_update, lru_add_drain_per_cpu [ 2670.121886] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2670.124618] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2670.127441] workqueue writeback: flags=0x4a [ 2670.129157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2670.131616] pending: wb_workfn, wb_workfn [ 2670.133365] workqueue xprtiod: flags=0xa [ 2670.134493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2670.136531] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2670.138803] workqueue ipv6_addrconf: flags=0x40008 [ 2670.140564] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2670.143389] pending: addrconf_verify_work [ 2670.145452] workqueue knbd0-recv: flags=0x1a [ 2670.147091] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2670.149502] in-flight: 655:recv_work [ 2670.151132] workqueue ptlrpc_pinger: flags=0x2 [ 2670.153092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2670.155758] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2670.157937] pool 8: cpus=0-3 flags=0x4 nice=0 hung=645s workers=4 idle: 36396 38 11188 [ 2670.160929] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2660s workers=2 idle: 658 [ 2672.054219] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2672.056871] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2672.109594] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2672.117213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2672.128230] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2672.133358] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2672.161170] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2672.164893] RAX: 0000000000002609 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2672.181318] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2672.189439] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2672.198109] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2672.205302] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2672.207534] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2672.211596] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2672.215427] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2672.221747] Call Trace: [ 2672.223849] [ 2672.225206] ? show_regs.cold.9+0x22/0x2f [ 2672.228572] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2672.238712] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2672.243140] ? watchdog+0x40/0x40 [ 2672.244491] ? hrtimer_interrupt+0x128/0x2c0 [ 2672.246468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2672.249094] ? apic_timer_interrupt+0xf/0x20 [ 2672.250364] [ 2672.251401] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2672.253398] do_raw_spin_lock+0x177/0x190 [ 2672.254629] _raw_spin_lock+0x21/0x30 [ 2672.256503] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2672.259382] ? do_raw_spin_unlock+0x75/0x190 [ 2672.261618] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2672.263978] ? do_wait_intr+0xf0/0xf0 [ 2672.265469] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2672.267338] mdc_close+0x53b/0xd70 [mdc] [ 2672.272743] lmv_close+0x18d/0x490 [lmv] [ 2672.275567] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2672.278630] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2672.286322] ll_md_close+0x274/0x800 [lustre] [ 2672.294424] ll_file_release+0x128/0xc50 [lustre] [ 2672.300275] __fput+0x10d/0x440 [ 2672.302944] ____fput+0x12/0x20 [ 2672.305906] task_work_run+0xc6/0x110 [ 2672.307566] exit_to_usermode_loop+0x1dd/0x1f0 [ 2672.309471] do_syscall_64+0x3d6/0x440 [ 2672.315649] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2672.323391] RIP: 0033:0x7fdc0afbfcd8 [ 2672.325425] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2672.336393] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2672.349608] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2672.356039] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2672.362750] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2672.370007] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2672.375614] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2688.051130] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2688.054926] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2688.059198] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2688.059279] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2688.059287] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2688.059294] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2688.059679] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2688.059689] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2688.059693] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2688.059697] RAX: 0000000000000a96 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2688.059700] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2688.059702] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2688.059704] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2688.059707] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2688.059710] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2688.059713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2688.059716] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2688.059724] Call Trace: [ 2688.059729] [ 2688.059734] ? show_regs.cold.9+0x22/0x2f [ 2688.059742] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2688.059748] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2688.059752] ? watchdog+0x40/0x40 [ 2688.059756] ? hrtimer_interrupt+0x128/0x2c0 [ 2688.059760] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2688.059767] ? apic_timer_interrupt+0xf/0x20 [ 2688.059772] [ 2688.059775] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2688.059779] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2688.059793] do_raw_spin_lock+0x177/0x190 [ 2688.065322] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [ 2688.085736] _raw_spin_lock+0x21/0x30 [ 2688.085757] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2688.086075] process_one_work+0x2c8/0x700 [ 2688.086082] worker_thread+0x296/0x6e0 [ 2688.086086] ? rescuer_thread+0x570/0x570 [ 2688.086089] kthread+0x1d1/0x200 [ 2688.086093] ? set_kthread_struct+0x70/0x70 [ 2688.086096] ret_from_fork+0x1f/0x30 [ 2688.235954] crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2688.243616] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2688.251544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2688.260859] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2688.264922] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2688.281619] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2688.287501] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2688.289889] RDX: 0000000000001208 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2688.293131] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2688.296117] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2688.299081] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2688.302532] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2688.306334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2688.309330] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2688.312569] Call Trace: [ 2688.313705] [ 2688.314671] ? show_regs.cold.9+0x22/0x2f [ 2688.316580] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2688.318697] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2688.320731] ? watchdog+0x40/0x40 [ 2688.322298] ? hrtimer_interrupt+0x128/0x2c0 [ 2688.324275] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2688.326615] ? apic_timer_interrupt+0xf/0x20 [ 2688.328587] [ 2688.329591] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2688.331895] do_raw_spin_lock+0x177/0x190 [ 2688.334338] _raw_spin_lock+0x21/0x30 [ 2688.335936] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2688.338675] ? do_raw_spin_unlock+0x75/0x190 [ 2688.340537] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2688.342919] ? do_wait_intr+0xf0/0xf0 [ 2688.347671] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2688.350589] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2688.353321] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2688.356247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2688.358856] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2688.361566] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2688.363632] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2688.365667] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2688.368185] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2688.370303] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2688.372925] lmv_intent_open+0x338/0x1150 [lmv] [ 2688.375813] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2688.378568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2688.384257] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2688.386964] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2688.390965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2688.393871] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2688.398671] ? do_raw_spin_unlock+0x75/0x190 [ 2688.401032] ? _raw_spin_unlock+0x12/0x30 [ 2688.403236] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2688.405479] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2688.414679] ll_atomic_open+0x36a/0xd70 [lustre] [ 2688.416365] lookup_open+0x356/0xb40 [ 2688.417520] do_last+0x313/0xfc0 [ 2688.418544] ? nd_jump_root+0xe5/0x160 [ 2688.419682] ? path_init+0x437/0x520 [ 2688.420839] path_openat+0xf7/0x500 [ 2688.421902] ? do_raw_spin_unlock+0x75/0x190 [ 2688.423159] do_filp_open+0x99/0x140 [ 2688.424333] ? getname_flags+0x6e/0x330 [ 2688.425522] ? __check_object_size+0xff/0x256 [ 2688.426842] ? do_raw_spin_unlock+0x75/0x190 [ 2688.428046] ? _raw_spin_unlock+0x12/0x30 [ 2688.429311] do_sys_openat2+0x2b4/0x410 [ 2688.430467] do_sys_open+0x73/0xa0 [ 2688.431451] __x64_sys_openat+0x24/0x30 [ 2688.432539] do_syscall_64+0xc1/0x440 [ 2688.433553] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2688.435949] RIP: 0033:0x7fdc0afbf332 [ 2688.437638] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2688.443638] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2688.446089] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2688.448664] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2688.450839] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2688.453383] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2688.455842] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2696.048249] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2696.062425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2696.153139] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2696.170968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2696.181321] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2696.183393] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2696.213332] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2696.219197] RAX: 0000000000002afd RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2696.226558] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2696.231025] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2696.241199] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2696.249622] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2696.259869] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2696.265207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2696.269126] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2696.281809] Call Trace: [ 2696.282715] [ 2696.283456] ? show_regs.cold.9+0x22/0x2f [ 2696.289095] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2696.295986] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2696.300664] ? watchdog+0x40/0x40 [ 2696.304847] ? hrtimer_interrupt+0x128/0x2c0 [ 2696.311073] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2696.319942] ? apic_timer_interrupt+0xf/0x20 [ 2696.322670] [ 2696.323654] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2696.329684] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2696.336170] ? dump_stack+0xf6/0x10e [ 2696.341715] do_raw_spin_lock+0x177/0x190 [ 2696.353934] _raw_spin_lock+0x21/0x30 [ 2696.357408] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2696.363551] obd_mod_free+0x46/0x140 [ptlrpc] [ 2696.368016] mdc_commit_open+0x91/0xb0 [mdc] [ 2696.369599] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2696.388657] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2696.397783] after_reply+0xe2c/0x1780 [ptlrpc] [ 2696.404148] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2696.409796] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2696.414754] ? do_wait_intr+0xf0/0xf0 [ 2696.417810] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2696.422792] kthread+0x1d1/0x200 [ 2696.425700] ? set_kthread_struct+0x70/0x70 [ 2696.430953] ret_from_fork+0x1f/0x30 [ 2700.055054] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2700.067323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2700.124685] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2700.141463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2700.150011] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2700.160710] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2700.189988] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2700.201719] RAX: 0000000000007f89 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2700.210130] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2700.218887] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2700.230018] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2700.235362] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2700.245354] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2700.249545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2700.257463] BUG: workqueue lockup - pool [ 2700.259057] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2700.259073] Call Trace: [ 2700.259080] [ 2700.259085] ? show_regs.cold.9+0x22/0x2f [ 2700.259100] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2700.259106] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2700.259110] ? watchdog+0x40/0x40 [ 2700.259125] ? hrtimer_interrupt+0x128/0x2c0 [ 2700.259129] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2700.259138] ? apic_timer_interrupt+0xf/0x20 [ 2700.259142] [ 2700.259144] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2700.259382] do_raw_spin_lock+0x177/0x190 [ 2700.259387] _raw_spin_lock+0x21/0x30 [ 2700.259393] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2700.259676] ? do_raw_spin_unlock+0x75/0x190 [ 2700.259683] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2700.259903] ? do_wait_intr+0xf0/0xf0 [ 2700.259911] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2700.261506] mdc_close+0x53b/0xd70 [mdc] [ 2700.261549] lmv_close+0x18d/0x490 [lmv] [ 2700.261596] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2700.261711] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2700.261804] ll_md_close+0x274/0x800 [lustre] [ 2700.261897] ll_file_release+0x128/0xc50 [lustre] [ 2700.263521] __fput+0x10d/0x440 [ 2700.263532] ____fput+0x12/0x20 [ 2700.263537] task_work_run+0xc6/0x110 [ 2700.263546] exit_to_usermode_loop+0x1dd/0x1f0 [ 2700.263553] do_syscall_64+0x3d6/0x440 [ 2700.263557] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2700.263567] RIP: 0033:0x7fdc0afbfcd8 [ 2700.263587] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2700.263592] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2700.263597] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2700.263599] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2700.263601] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2700.263603] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2700.263605] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2700.547993] cpus=0-3 flags=0x4 nice=0 stuck for 675s! [ 2700.552784] Showing busy workqueues and worker pools: [ 2700.554534] workqueue events: flags=0x0 [ 2700.562439] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2700.568412] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2700.572939] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2700.575401] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2700.593031] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2700.599103] pending: kfree_rcu_monitor [ 2700.602840] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2700.610551] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2700.621532] workqueue events_long: flags=0x0 [ 2700.623130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2700.629996] pending: lu_objects_flush [obdclass] [ 2700.634955] workqueue events_unbound: flags=0x2 [ 2700.639801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2700.646306] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2700.659791] workqueue events_power_efficient: flags=0x80 [ 2700.666738] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2700.671961] pending: neigh_periodic_work, check_lifetime [ 2700.679270] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2700.682929] pending: do_cache_clean, neigh_periodic_work [ 2700.684922] workqueue mm_percpu_wq: flags=0x8 [ 2700.686317] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2700.690700] pending: lru_add_drain_per_cpu [ 2700.693810] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2700.697371] pending: vmstat_update, lru_add_drain_per_cpu [ 2700.711287] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2700.713370] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2700.724889] workqueue writeback: flags=0x4a [ 2700.726355] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2700.743192] pending: wb_workfn, wb_workfn [ 2700.750842] workqueue xprtiod: flags=0xa [ 2700.755977] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2700.764025] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2700.773526] workqueue ipv6_addrconf: flags=0x40008 [ 2700.779037] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2700.790026] pending: addrconf_verify_work [ 2700.794030] workqueue knbd0-recv: flags=0x1a [ 2700.803800] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2700.813761] in-flight: 655:recv_work [ 2700.816659] workqueue ptlrpc_pinger: flags=0x2 [ 2700.823637] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2700.825869] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2700.844420] pool 8: cpus=0-3 flags=0x4 nice=0 hung=675s workers=4 idle: 36396 38 11188 [ 2700.852040] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2691s workers=2 idle: 658 [ 2716.051041] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2716.052643] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2716.058660] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs [ 2716.067571] Modules linked in: [ 2716.067573] ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel [ 2716.082142] lustre(O) osp(O) [ 2716.083764] ata_piix [ 2716.086420] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [ 2716.088159] ghash_clmulni_intel [ 2716.089216] osc(O) [ 2716.094787] libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2716.094810] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2716.094817] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2716.094823] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2716.096409] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 [ 2716.097013] [ 2716.097016] RIP: 0010:__pv_queued_spin_lock_slowpath+0x315/0x6d0 [ 2716.102126] pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2716.102152] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2716.102159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2716.102162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2716.102176] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2716.102180] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2716.102185] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2716.102187] RDX: 0000000000006bc2 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2716.102189] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2716.102192] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2716.102194] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2716.102197] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2716.102200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2716.102202] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2716.102210] Call Trace: [ 2716.102215] [ 2716.102220] ? show_regs.cold.9+0x22/0x2f [ 2716.102228] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2716.102232] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2716.102236] ? watchdog+0x40/0x40 [ 2716.102239] ? hrtimer_interrupt+0x128/0x2c0 [ 2716.102241] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2716.102465] ? apic_timer_interrupt+0xf/0x20 [ 2716.102470] [ 2716.102474] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2716.102480] do_raw_spin_lock+0x177/0x190 [ 2716.102484] _raw_spin_lock+0x21/0x30 [ 2716.102492] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2716.102788] ? do_raw_spin_unlock+0x75/0x190 [ 2716.102793] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2716.109603] Code: 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 <8b> 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 eb e0 48 [ 2716.109617] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2716.109625] RAX: 0000000000000cf4 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2716.109632] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2716.109633] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2716.109635] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2716.109637] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2716.109640] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2716.109643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2716.109645] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2716.109653] Call Trace: [ 2716.109658] [ 2716.109664] ? show_regs.cold.9+0x22/0x2f [ 2716.109676] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2716.109684] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2716.109690] ? watchdog+0x40/0x40 [ 2716.109694] ? hrtimer_interrupt+0x128/0x2c0 [ 2716.109765] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2716.109774] ? apic_timer_interrupt+0xf/0x20 [ 2716.109778] [ 2716.109781] ? __pv_queued_spin_lock_slowpath+0x315/0x6d0 [ 2716.109787] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2716.109794] do_raw_spin_lock+0x177/0x190 [ 2716.109799] _raw_spin_lock+0x21/0x30 [ 2716.109810] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2716.110454] process_one_work+0x2c8/0x700 [ 2716.110462] worker_thread+0x296/0x6e0 [ 2716.110467] ? rescuer_thread+0x570/0x570 [ 2716.110470] kthread+0x1d1/0x200 [ 2716.110475] ? set_kthread_struct+0x70/0x70 [ 2716.110479] ret_from_fork+0x1f/0x30 [ 2716.395505] ? do_wait_intr+0xf0/0xf0 [ 2716.396707] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2716.399263] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2716.401592] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2716.404412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2716.406624] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2716.408763] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2716.414677] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2716.423625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2716.430260] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2716.434669] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2716.442079] lmv_intent_open+0x338/0x1150 [lmv] [ 2716.445448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2716.448616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2716.453131] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2716.456595] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2716.462657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2716.469780] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2716.472106] ? do_raw_spin_unlock+0x75/0x190 [ 2716.474327] ? _raw_spin_unlock+0x12/0x30 [ 2716.476660] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2716.479832] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2716.481284] ll_atomic_open+0x36a/0xd70 [lustre] [ 2716.484485] lookup_open+0x356/0xb40 [ 2716.488267] do_last+0x313/0xfc0 [ 2716.489661] ? nd_jump_root+0xe5/0x160 [ 2716.492370] ? path_init+0x437/0x520 [ 2716.497624] path_openat+0xf7/0x500 [ 2716.503157] ? do_raw_spin_unlock+0x75/0x190 [ 2716.506797] do_filp_open+0x99/0x140 [ 2716.510759] ? getname_flags+0x6e/0x330 [ 2716.512956] ? __check_object_size+0xff/0x256 [ 2716.521474] ? do_raw_spin_unlock+0x75/0x190 [ 2716.522831] ? _raw_spin_unlock+0x12/0x30 [ 2716.528870] do_sys_openat2+0x2b4/0x410 [ 2716.532735] do_sys_open+0x73/0xa0 [ 2716.534410] __x64_sys_openat+0x24/0x30 [ 2716.535646] do_syscall_64+0xc1/0x440 [ 2716.538216] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2716.540405] RIP: 0033:0x7fdc0afbf332 [ 2716.542305] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2716.555621] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2716.561380] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2716.564652] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2716.573587] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2716.575552] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2716.584598] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2728.051194] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2728.056643] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2728.057871] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2728.083783] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2728.083800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2728.083803] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2728.083817] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2728.083822] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2728.083829] RAX: 0000000000007d97 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2728.083831] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2728.083833] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2728.083835] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2728.083838] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2728.083841] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2728.083843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2728.083846] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2728.083854] Call Trace: [ 2728.083860] [ 2728.083865] ? show_regs.cold.9+0x22/0x2f [ 2728.083883] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2728.083890] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2728.083894] ? watchdog+0x40/0x40 [ 2728.083898] ? hrtimer_interrupt+0x128/0x2c0 [ 2728.083902] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2728.083910] ? apic_timer_interrupt+0xf/0x20 [ 2728.083915] [ 2728.083918] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2728.083923] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2728.083927] ? dump_stack+0xf6/0x10e [ 2728.083932] do_raw_spin_lock+0x177/0x190 [ 2728.083937] _raw_spin_lock+0x21/0x30 [ 2728.083945] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2728.084302] obd_mod_free+0x46/0x140 [ptlrpc] [ 2728.084556] mdc_commit_open+0x91/0xb0 [mdc] [ 2728.084604] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2728.084851] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2728.085130] after_reply+0xe2c/0x1780 [ptlrpc] [ 2728.085405] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2728.091371] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2728.091449] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2728.091455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2728.091458] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2728.091472] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2728.091476] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2728.091481] RAX: 0000000000003910 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2728.091483] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2728.091485] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2728.091487] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2728.091490] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2728.091493] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2728.091496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2728.091498] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2728.091506] Call Trace: [ 2728.091511] [ 2728.091516] ? show_regs.cold.9+0x22/0x2f [ 2728.091539] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2728.091545] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2728.091549] ? watchdog+0x40/0x40 [ 2728.091553] ? hrtimer_interrupt+0x128/0x2c0 [ 2728.091557] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2728.091567] ? apic_timer_interrupt+0xf/0x20 [ 2728.091572] [ 2728.091575] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2728.091581] do_raw_spin_lock+0x177/0x190 [ 2728.091586] _raw_spin_lock+0x21/0x30 [ 2728.091591] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2728.092110] ? do_raw_spin_unlock+0x75/0x190 [ 2728.092184] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2728.092446] ? do_wait_intr+0xf0/0xf0 [ 2728.092482] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2728.098841] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2728.099255] ? do_wait_intr+0xf0/0xf0 [ 2728.099267] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2728.099551] kthread+0x1d1/0x200 [ 2728.099558] ? set_kthread_struct+0x70/0x70 [ 2728.099562] ret_from_fork+0x1f/0x30 [ 2728.296447] mdc_close+0x53b/0xd70 [mdc] [ 2728.297657] lmv_close+0x18d/0x490 [lmv] [ 2728.300559] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2728.304027] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2728.306436] ll_md_close+0x274/0x800 [lustre] [ 2728.310222] ll_file_release+0x128/0xc50 [lustre] [ 2728.313216] __fput+0x10d/0x440 [ 2728.316335] ____fput+0x12/0x20 [ 2728.317553] task_work_run+0xc6/0x110 [ 2728.322712] exit_to_usermode_loop+0x1dd/0x1f0 [ 2728.324001] do_syscall_64+0x3d6/0x440 [ 2728.325599] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2728.329542] RIP: 0033:0x7fdc0afbfcd8 [ 2728.333138] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2728.343917] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2728.348926] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2728.352985] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2728.358111] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2728.363733] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2728.366253] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2730.976129] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 706s! [ 2730.981922] Showing busy workqueues and worker pools: [ 2730.987783] workqueue events: flags=0x0 [ 2730.995136] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2731.003831] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2731.006524] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2731.008838] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2731.011784] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2731.019093] pending: kfree_rcu_monitor [ 2731.022674] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2731.025878] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2731.030561] workqueue events_long: flags=0x0 [ 2731.031918] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2731.042416] pending: lu_objects_flush [obdclass] [ 2731.044468] workqueue events_unbound: flags=0x2 [ 2731.046396] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2731.051755] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2731.060618] workqueue events_power_efficient: flags=0x80 [ 2731.064381] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2731.067347] pending: neigh_periodic_work, check_lifetime [ 2731.069203] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2731.072717] pending: do_cache_clean, neigh_periodic_work [ 2731.075352] workqueue mm_percpu_wq: flags=0x8 [ 2731.077498] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2731.080170] pending: lru_add_drain_per_cpu [ 2731.083977] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2731.087796] pending: vmstat_update, lru_add_drain_per_cpu [ 2731.094241] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2731.095995] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2731.103992] workqueue writeback: flags=0x4a [ 2731.108449] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2731.113683] pending: wb_workfn, wb_workfn [ 2731.120292] workqueue xprtiod: flags=0xa [ 2731.124197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2731.131323] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2731.141539] workqueue ipv6_addrconf: flags=0x40008 [ 2731.146267] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2731.150618] pending: addrconf_verify_work [ 2731.156170] workqueue knbd0-recv: flags=0x1a [ 2731.157759] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2731.162651] in-flight: 655:recv_work [ 2731.164557] workqueue ptlrpc_pinger: flags=0x2 [ 2731.168685] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2731.173747] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2731.177528] pool 8: cpus=0-3 flags=0x4 nice=0 hung=706s workers=4 idle: 36396 38 11188 [ 2731.180596] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2721s workers=2 idle: 658 [ 2744.054316] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2744.055055] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2744.062364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2744.063138] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2744.063146] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2744.063148] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2744.063162] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2744.063166] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2744.063170] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2744.063173] RDX: 0000000000002729 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2744.063176] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2744.063178] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2744.063180] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2744.063182] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2744.063185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2744.063187] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2744.063197] Call Trace: [ 2744.063203] [ 2744.063208] ? show_regs.cold.9+0x22/0x2f [ 2744.063217] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2744.063224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2744.063228] ? watchdog+0x40/0x40 [ 2744.063233] ? hrtimer_interrupt+0x128/0x2c0 [ 2744.063237] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2744.063246] ? apic_timer_interrupt+0xf/0x20 [ 2744.063250] [ 2744.063253] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2744.063258] do_raw_spin_lock+0x177/0x190 [ 2744.063262] _raw_spin_lock+0x21/0x30 [ 2744.063268] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2744.064121] ? do_raw_spin_unlock+0x75/0x190 [ 2744.064128] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2744.064345] ? do_wait_intr+0xf0/0xf0 [ 2744.064354] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2744.064911] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2744.065144] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2744.065368] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2744.065604] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2744.065772] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2744.065814] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2744.065842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2744.065925] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2744.066150] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2744.066178] lmv_intent_open+0x338/0x1150 [lmv] [ 2744.066204] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2744.066305] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2744.066390] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2744.066854] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2744.066944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2744.067030] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2744.067139] ? do_raw_spin_unlock+0x75/0x190 [ 2744.067146] ? _raw_spin_unlock+0x12/0x30 [ 2744.067152] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2744.067178] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2744.067202] ll_atomic_open+0x36a/0xd70 [lustre] [ 2744.074109] Modules linked in: lustre(O) [ 2744.100514] lookup_open+0x356/0xb40 [ 2744.114153] osp(O) [ 2744.121925] do_last+0x313/0xfc0 [ 2744.121943] ? nd_jump_root+0xe5/0x160 [ 2744.121948] ? path_init+0x437/0x520 [ 2744.121953] path_openat+0xf7/0x500 [ 2744.121959] ? do_raw_spin_unlock+0x75/0x190 [ 2744.121966] do_filp_open+0x99/0x140 [ 2744.121973] ? getname_flags+0x6e/0x330 [ 2744.121978] ? __check_object_size+0xff/0x256 [ 2744.121985] ? do_raw_spin_unlock+0x75/0x190 [ 2744.121989] ? _raw_spin_unlock+0x12/0x30 [ 2744.121996] do_sys_openat2+0x2b4/0x410 [ 2744.122001] do_sys_open+0x73/0xa0 [ 2744.122005] __x64_sys_openat+0x24/0x30 [ 2744.122009] do_syscall_64+0xc1/0x440 [ 2744.122016] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2744.122022] RIP: 0033:0x7fdc0afbf332 [ 2744.122031] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2744.122037] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2744.122043] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2744.122046] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2744.122049] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2744.122051] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2744.122054] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2744.434760] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2744.469934] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2744.480210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2744.483474] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2744.488939] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2744.494355] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2744.513669] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2744.521264] RAX: 00000000000067d5 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2744.523242] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2744.530338] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2744.537789] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2744.541573] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2744.547486] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2744.562417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2744.569190] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2744.580161] Call Trace: [ 2744.586686] [ 2744.590475] ? show_regs.cold.9+0x22/0x2f [ 2744.597934] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2744.601336] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2744.610051] ? watchdog+0x40/0x40 [ 2744.613236] ? hrtimer_interrupt+0x128/0x2c0 [ 2744.621154] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2744.630047] ? apic_timer_interrupt+0xf/0x20 [ 2744.635778] [ 2744.636781] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2744.643218] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2744.650333] do_raw_spin_lock+0x177/0x190 [ 2744.655333] _raw_spin_lock+0x21/0x30 [ 2744.661930] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2744.676586] process_one_work+0x2c8/0x700 [ 2744.680505] worker_thread+0x296/0x6e0 [ 2744.686460] ? rescuer_thread+0x570/0x570 [ 2744.692245] kthread+0x1d1/0x200 [ 2744.698414] ? set_kthread_struct+0x70/0x70 [ 2744.703610] ret_from_fork+0x1f/0x30 [ 2756.048029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2756.052745] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2756.054952] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2756.075372] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2756.075386] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2756.075389] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2756.075402] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2756.075407] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2756.075412] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2756.075415] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2756.075417] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2756.075419] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2756.075422] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2756.075425] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2756.075428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2756.075430] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2756.075437] Call Trace: [ 2756.075443] [ 2756.075448] ? show_regs.cold.9+0x22/0x2f [ 2756.075458] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2756.075464] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2756.075468] ? watchdog+0x40/0x40 [ 2756.075472] ? hrtimer_interrupt+0x128/0x2c0 [ 2756.075475] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2756.075482] ? apic_timer_interrupt+0xf/0x20 [ 2756.075486] [ 2756.075489] ? native_safe_halt+0x2e/0x50 [ 2756.075492] kvm_wait+0xcd/0xe0 [ 2756.075500] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2756.075507] ? dump_stack+0xf6/0x10e [ 2756.075513] do_raw_spin_lock+0x177/0x190 [ 2756.075518] _raw_spin_lock+0x21/0x30 [ 2756.075525] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2756.075989] obd_mod_free+0x46/0x140 [ptlrpc] [ 2756.076388] mdc_commit_open+0x91/0xb0 [mdc] [ 2756.076427] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2756.076643] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2756.076972] after_reply+0xe2c/0x1780 [ptlrpc] [ 2756.077254] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2756.077493] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2756.082004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2756.082094] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2756.082101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2756.082104] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2756.082120] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2756.082123] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2756.082127] RAX: 00000000000034b9 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2756.082129] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2756.082131] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2756.082133] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2756.082136] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2756.082138] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2756.082141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2756.082143] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2756.082152] Call Trace: [ 2756.082157] [ 2756.082163] ? show_regs.cold.9+0x22/0x2f [ 2756.082174] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2756.082180] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2756.087884] ? do_wait_intr+0xf0/0xf0 [ 2756.087907] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2756.088476] kthread+0x1d1/0x200 [ 2756.088483] ? set_kthread_struct+0x70/0x70 [ 2756.088487] ret_from_fork+0x1f/0x30 [ 2756.488025] ? watchdog+0x40/0x40 [ 2756.490304] ? hrtimer_interrupt+0x128/0x2c0 [ 2756.493844] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2756.498513] ? apic_timer_interrupt+0xf/0x20 [ 2756.502009] [ 2756.502921] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2756.507398] do_raw_spin_lock+0x177/0x190 [ 2756.511401] _raw_spin_lock+0x21/0x30 [ 2756.514032] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2756.517047] ? do_raw_spin_unlock+0x75/0x190 [ 2756.519333] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2756.524460] ? do_wait_intr+0xf0/0xf0 [ 2756.525713] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2756.532006] mdc_close+0x53b/0xd70 [mdc] [ 2756.533347] lmv_close+0x18d/0x490 [lmv] [ 2756.534470] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2756.539486] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2756.543434] ll_md_close+0x274/0x800 [lustre] [ 2756.546360] ll_file_release+0x128/0xc50 [lustre] [ 2756.549290] __fput+0x10d/0x440 [ 2756.551525] ____fput+0x12/0x20 [ 2756.555079] task_work_run+0xc6/0x110 [ 2756.557251] exit_to_usermode_loop+0x1dd/0x1f0 [ 2756.559144] do_syscall_64+0x3d6/0x440 [ 2756.561736] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2756.564029] RIP: 0033:0x7fdc0afbfcd8 [ 2756.565513] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2756.578233] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2756.581518] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2756.584443] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2756.596961] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2756.602079] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2756.607334] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2761.695631] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 736s! [ 2761.700839] Showing busy workqueues and worker pools: [ 2761.705656] workqueue events: flags=0x0 [ 2761.710913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2761.717486] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2761.722960] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2761.730281] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2761.736432] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2761.742244] pending: kfree_rcu_monitor [ 2761.745266] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2761.750012] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2761.763156] workqueue events_long: flags=0x0 [ 2761.766252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2761.773740] pending: lu_objects_flush [obdclass] [ 2761.780365] workqueue events_unbound: flags=0x2 [ 2761.784460] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2761.789250] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2761.795028] workqueue events_power_efficient: flags=0x80 [ 2761.798876] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2761.801212] pending: neigh_periodic_work, check_lifetime [ 2761.804546] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2761.811476] pending: do_cache_clean, neigh_periodic_work [ 2761.817448] workqueue mm_percpu_wq: flags=0x8 [ 2761.821753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2761.824775] pending: lru_add_drain_per_cpu [ 2761.827240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2761.832533] pending: vmstat_update, lru_add_drain_per_cpu [ 2761.835416] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2761.839151] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2761.844034] workqueue writeback: flags=0x4a [ 2761.847864] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2761.855550] pending: wb_workfn, wb_workfn [ 2761.863801] workqueue xprtiod: flags=0xa [ 2761.865257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2761.871553] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2761.875706] workqueue ipv6_addrconf: flags=0x40008 [ 2761.878697] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2761.884735] pending: addrconf_verify_work [ 2761.889776] workqueue knbd0-recv: flags=0x1a [ 2761.893475] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2761.899564] in-flight: 655:recv_work [ 2761.902408] workqueue ptlrpc_pinger: flags=0x2 [ 2761.911017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2761.916759] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2761.923477] pool 8: cpus=0-3 flags=0x4 nice=0 hung=736s workers=4 idle: 36396 38 11188 [ 2761.928988] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2752s workers=2 idle: 658 [ 2772.052773] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2772.053248] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2772.057476] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2772.059183] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2772.059190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2772.059194] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2772.059208] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2772.059212] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2772.059217] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2772.059219] RDX: 0000000000004eaf RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2772.059222] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2772.059224] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2772.059226] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2772.059230] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2772.059232] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2772.059234] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2772.059243] Call Trace: [ 2772.059248] [ 2772.059253] ? show_regs.cold.9+0x22/0x2f [ 2772.059262] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2772.059268] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2772.059272] ? watchdog+0x40/0x40 [ 2772.059276] ? hrtimer_interrupt+0x128/0x2c0 [ 2772.059279] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2772.059289] ? apic_timer_interrupt+0xf/0x20 [ 2772.059294] [ 2772.059297] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2772.059302] do_raw_spin_lock+0x177/0x190 [ 2772.059307] _raw_spin_lock+0x21/0x30 [ 2772.059323] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2772.059604] ? do_raw_spin_unlock+0x75/0x190 [ 2772.059610] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2772.059828] ? do_wait_intr+0xf0/0xf0 [ 2772.059836] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2772.060104] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2772.063570] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2772.063657] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2772.063665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2772.063672] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2772.087097] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2772.087468] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2772.088010] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2772.088074] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2772.088119] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2772.088159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2772.088245] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2772.088731] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2772.088763] lmv_intent_open+0x338/0x1150 [lmv] [ 2772.088794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2772.088892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2772.088981] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2772.089005] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2772.089091] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2772.089169] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2772.089286] ? do_raw_spin_unlock+0x75/0x190 [ 2772.089297] ? _raw_spin_unlock+0x12/0x30 [ 2772.089306] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2772.089358] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2772.089389] ll_atomic_open+0x36a/0xd70 [lustre] [ 2772.089649] lookup_open+0x356/0xb40 [ 2772.089663] do_last+0x313/0xfc0 [ 2772.089670] ? nd_jump_root+0xe5/0x160 [ 2772.089675] ? path_init+0x437/0x520 [ 2772.089681] path_openat+0xf7/0x500 [ 2772.089687] ? do_raw_spin_unlock+0x75/0x190 [ 2772.089694] do_filp_open+0x99/0x140 [ 2772.089702] ? getname_flags+0x6e/0x330 [ 2772.089709] ? __check_object_size+0xff/0x256 [ 2772.089718] ? do_raw_spin_unlock+0x75/0x190 [ 2772.089723] ? _raw_spin_unlock+0x12/0x30 [ 2772.089728] do_sys_openat2+0x2b4/0x410 [ 2772.089733] do_sys_open+0x73/0xa0 [ 2772.089739] __x64_sys_openat+0x24/0x30 [ 2772.089743] do_syscall_64+0xc1/0x440 [ 2772.089751] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2772.089759] RIP: 0033:0x7fdc0afbf332 [ 2772.089769] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2772.089774] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2772.089780] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2772.089782] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2772.089784] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2772.089786] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2772.089789] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2772.441869] [ 2772.442652] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2772.446058] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2772.459204] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2772.469648] RAX: 0000000000004f59 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2772.477529] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2772.485552] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2772.491010] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2772.497260] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2772.502533] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2772.507635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2772.510039] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2772.512893] Call Trace: [ 2772.513699] [ 2772.514497] ? show_regs.cold.9+0x22/0x2f [ 2772.515870] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2772.517647] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2772.519319] ? watchdog+0x40/0x40 [ 2772.520629] ? hrtimer_interrupt+0x128/0x2c0 [ 2772.522204] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2772.524086] ? apic_timer_interrupt+0xf/0x20 [ 2772.525569] [ 2772.526422] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2772.528159] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2772.531871] do_raw_spin_lock+0x177/0x190 [ 2772.533141] _raw_spin_lock+0x21/0x30 [ 2772.535806] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2772.538493] process_one_work+0x2c8/0x700 [ 2772.540267] worker_thread+0x296/0x6e0 [ 2772.541963] ? rescuer_thread+0x570/0x570 [ 2772.543326] kthread+0x1d1/0x200 [ 2772.547635] ? set_kthread_struct+0x70/0x70 [ 2772.552404] ret_from_fork+0x1f/0x30 [ 2784.054027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2784.067051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2784.114683] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2784.127075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2784.138939] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2784.143284] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2784.173732] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2784.184370] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2784.202604] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2784.211851] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2784.214443] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2784.224797] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2784.230869] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2784.234979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2784.238977] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2784.242330] Call Trace: [ 2784.243485] [ 2784.245499] ? show_regs.cold.9+0x22/0x2f [ 2784.249437] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2784.252275] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2784.258213] ? watchdog+0x40/0x40 [ 2784.261939] ? hrtimer_interrupt+0x128/0x2c0 [ 2784.266287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2784.274150] ? apic_timer_interrupt+0xf/0x20 [ 2784.275718] [ 2784.277461] ? native_safe_halt+0x2e/0x50 [ 2784.281202] kvm_wait+0xcd/0xe0 [ 2784.284208] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2784.288434] do_raw_spin_lock+0x177/0x190 [ 2784.290281] _raw_spin_lock+0x21/0x30 [ 2784.292893] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2784.295920] ? do_raw_spin_unlock+0x75/0x190 [ 2784.299324] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2784.304584] ? do_wait_intr+0xf0/0xf0 [ 2784.307087] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2784.310248] mdc_close+0x53b/0xd70 [mdc] [ 2784.312250] lmv_close+0x18d/0x490 [lmv] [ 2784.314088] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2784.317107] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2784.319863] ll_md_close+0x274/0x800 [lustre] [ 2784.322823] ll_file_release+0x128/0xc50 [lustre] [ 2784.326088] __fput+0x10d/0x440 [ 2784.327584] ____fput+0x12/0x20 [ 2784.329344] task_work_run+0xc6/0x110 [ 2784.331383] exit_to_usermode_loop+0x1dd/0x1f0 [ 2784.334864] do_syscall_64+0x3d6/0x440 [ 2784.337249] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2784.344377] RIP: 0033:0x7fdc0afbfcd8 [ 2784.349843] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2784.363421] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2784.367991] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2784.372669] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2784.380244] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2784.385891] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2784.390112] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2788.048024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2788.051788] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2788.084313] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2788.089141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2788.094314] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2788.095969] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2788.111796] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2788.117437] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2788.120568] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2788.123663] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2788.126449] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2788.131070] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2788.133492] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2788.136449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2788.139563] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2788.145519] Call Trace: [ 2788.147135] [ 2788.147956] ? show_regs.cold.9+0x22/0x2f [ 2788.149856] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2788.151994] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2788.155225] ? watchdog+0x40/0x40 [ 2788.157421] ? hrtimer_interrupt+0x128/0x2c0 [ 2788.158785] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2788.163887] ? apic_timer_interrupt+0xf/0x20 [ 2788.167965] [ 2788.168673] ? native_safe_halt+0x2e/0x50 [ 2788.169821] kvm_wait+0xcd/0xe0 [ 2788.170745] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2788.177306] ? dump_stack+0xf6/0x10e [ 2788.180992] do_raw_spin_lock+0x177/0x190 [ 2788.182506] _raw_spin_lock+0x21/0x30 [ 2788.186844] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2788.197251] obd_mod_free+0x46/0x140 [ptlrpc] [ 2788.201376] mdc_commit_open+0x91/0xb0 [mdc] [ 2788.205958] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2788.208870] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2788.214259] after_reply+0xe2c/0x1780 [ptlrpc] [ 2788.216483] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2788.221072] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2788.225943] ? do_wait_intr+0xf0/0xf0 [ 2788.243298] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2788.247230] kthread+0x1d1/0x200 [ 2788.248839] ? set_kthread_struct+0x70/0x70 [ 2788.250484] ret_from_fork+0x1f/0x30 [ 2792.415028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 767s! [ 2792.419274] Showing busy workqueues and worker pools: [ 2792.421851] workqueue events: flags=0x0 [ 2792.423498] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2792.426860] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2792.429423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2792.432315] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2792.436811] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2792.443232] pending: kfree_rcu_monitor [ 2792.449355] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2792.456087] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2792.461586] workqueue events_long: flags=0x0 [ 2792.463737] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2792.466991] pending: lu_objects_flush [obdclass] [ 2792.469573] workqueue events_unbound: flags=0x2 [ 2792.471869] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2792.475043] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2792.479850] workqueue events_power_efficient: flags=0x80 [ 2792.482083] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2792.487630] pending: neigh_periodic_work, check_lifetime [ 2792.492217] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2792.494184] pending: do_cache_clean, neigh_periodic_work [ 2792.497765] workqueue mm_percpu_wq: flags=0x8 [ 2792.500867] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2792.503366] pending: lru_add_drain_per_cpu [ 2792.505700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2792.509581] pending: vmstat_update, lru_add_drain_per_cpu [ 2792.513021] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2792.520327] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2792.523444] workqueue writeback: flags=0x4a [ 2792.525048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2792.527978] pending: wb_workfn, wb_workfn [ 2792.530825] workqueue xprtiod: flags=0xa [ 2792.532470] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2792.537945] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2792.541864] workqueue ipv6_addrconf: flags=0x40008 [ 2792.544375] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2792.547649] pending: addrconf_verify_work [ 2792.554131] workqueue knbd0-recv: flags=0x1a [ 2792.558743] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2792.566291] in-flight: 655:recv_work [ 2792.569762] workqueue ptlrpc_pinger: flags=0x2 [ 2792.578691] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2792.585504] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2792.589501] pool 8: cpus=0-3 flags=0x4 nice=0 hung=767s workers=4 idle: 36396 38 11188 [ 2792.593869] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2783s workers=2 idle: 658 [ 2799.360378] rcu: INFO: rcu_sched self-detected stall on CPU [ 2799.364322] rcu: 0-...!: (446847 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 2799.378623] (t=780017 jiffies g=270761 q=2874) [ 2799.380692] rcu: rcu_sched kthread starved for 774949 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 2799.389194] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2799.394070] rcu: RCU grace-period kthread stack dump: [ 2799.409607] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2799.419902] Call Trace: [ 2799.422080] __schedule+0x351/0xcb0 [ 2799.424800] schedule+0xc0/0x180 [ 2799.427634] schedule_timeout+0xb4/0x190 [ 2799.429610] ? do_raw_spin_unlock+0x75/0x190 [ 2799.431870] ? __next_timer_interrupt+0x160/0x160 [ 2799.438426] ? prepare_to_swait_event+0xf3/0x140 [ 2799.440135] rcu_gp_kthread+0x8dd/0x15b0 [ 2799.442733] ? force_qs_rnp+0x2c0/0x2c0 [ 2799.444232] kthread+0x1d1/0x200 [ 2799.449021] ? set_kthread_struct+0x70/0x70 [ 2799.450628] ret_from_fork+0x1f/0x30 [ 2799.456874] rcu: Stack dump where RCU GP kthread last ran: [ 2799.460697] Sending NMI from CPU 0 to CPUs 2: [ 2799.469128] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 2.893 msecs [ 2799.470576] NMI backtrace for cpu 2 [ 2799.470582] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2799.470585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2799.470586] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2799.470587] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2799.470589] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 [ 2799.470592] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2799.470593] RDX: 0000000000000fd3 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2799.470595] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2799.470596] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2799.470597] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2799.470598] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2799.470599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2799.470600] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2799.470601] Call Trace: [ 2799.470602] [ 2799.470602] ? show_regs.cold.9+0x22/0x2f [ 2799.470603] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2799.470605] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2799.470606] ? nmi_handle+0xbd/0x210 [ 2799.470607] ? default_do_nmi+0x4b/0x1b0 [ 2799.470608] ? do_nmi+0x280/0x380 [ 2799.470609] ? end_repeat_nmi+0x16/0x1f [ 2799.470610] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2799.470611] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2799.470612] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2799.470613] [ 2799.470614] do_raw_spin_lock+0x177/0x190 [ 2799.470615] _raw_spin_lock+0x21/0x30 [ 2799.470616] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2799.470617] ? do_raw_spin_unlock+0x75/0x190 [ 2799.470618] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2799.470619] ? do_wait_intr+0xf0/0xf0 [ 2799.470620] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2799.470621] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2799.470623] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2799.470624] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2799.470625] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2799.470626] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2799.470627] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2799.470628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2799.470629] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2799.470630] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2799.470631] lmv_intent_open+0x338/0x1150 [lmv] [ 2799.470632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2799.470633] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2799.470634] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2799.470635] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2799.470636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2799.470637] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2799.470638] ? do_raw_spin_unlock+0x75/0x190 [ 2799.470639] ? _raw_spin_unlock+0x12/0x30 [ 2799.470640] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2799.470641] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2799.470642] ll_atomic_open+0x36a/0xd70 [lustre] [ 2799.470644] lookup_open+0x356/0xb40 [ 2799.470645] do_last+0x313/0xfc0 [ 2799.470646] ? nd_jump_root+0xe5/0x160 [ 2799.470647] ? path_init+0x437/0x520 [ 2799.470648] path_openat+0xf7/0x500 [ 2799.470649] ? do_raw_spin_unlock+0x75/0x190 [ 2799.470650] do_filp_open+0x99/0x140 [ 2799.470651] ? getname_flags+0x6e/0x330 [ 2799.470652] ? __check_object_size+0xff/0x256 [ 2799.470653] ? do_raw_spin_unlock+0x75/0x190 [ 2799.470654] ? _raw_spin_unlock+0x12/0x30 [ 2799.470656] do_sys_openat2+0x2b4/0x410 [ 2799.470656] do_sys_open+0x73/0xa0 [ 2799.470657] __x64_sys_openat+0x24/0x30 [ 2799.470658] do_syscall_64+0xc1/0x440 [ 2799.470659] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2799.470661] RIP: 0033:0x7fdc0afbf332 [ 2799.470662] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2799.470664] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2799.470667] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2799.470668] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2799.470669] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2799.470671] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2799.470672] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2799.470677] NMI backtrace for cpu 0 [ 2799.745205] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2799.750090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2799.759352] Call Trace: [ 2799.764049] [ 2799.765822] ? dump_stack+0xbb/0x10e [ 2799.767669] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2799.771907] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2799.776643] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2799.781125] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2799.785224] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2799.789416] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2799.792111] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2799.795698] ? update_process_times+0x6d/0xc0 [ 2799.801703] ? tick_sched_handle.isra.18+0x27/0x90 [ 2799.806407] ? tick_sched_timer+0x53/0xb0 [ 2799.810840] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2799.813981] ? tick_sched_handle.isra.18+0x90/0x90 [ 2799.816764] ? hrtimer_interrupt+0x128/0x2c0 [ 2799.819286] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2799.821617] ? apic_timer_interrupt+0xf/0x20 [ 2799.826757] [ 2799.831232] ? native_safe_halt+0x2e/0x50 [ 2799.832562] ? kvm_wait+0xcd/0xe0 [ 2799.833509] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2799.841359] ? dump_stack+0xf6/0x10e [ 2799.844937] ? do_raw_spin_lock+0x177/0x190 [ 2799.846942] ? _raw_spin_lock+0x21/0x30 [ 2799.850884] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2799.858462] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2799.863559] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2799.869078] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2799.873841] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2799.876745] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2799.884654] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2799.890559] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2799.894722] ? do_wait_intr+0xf0/0xf0 [ 2799.895802] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2799.901641] ? kthread+0x1d1/0x200 [ 2799.903228] ? set_kthread_struct+0x70/0x70 [ 2799.904993] ? ret_from_fork+0x1f/0x30 [ 2800.051170] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2800.053797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2800.072926] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2800.078091] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2800.080855] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2800.084122] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2800.086110] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2800.092748] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2800.097100] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2800.101160] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2800.105405] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2800.109062] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2800.111274] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2800.114231] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2800.118318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2800.121307] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2800.124872] Call Trace: [ 2800.125633] [ 2800.126244] ? show_regs.cold.9+0x22/0x2f [ 2800.127374] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2800.129630] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2800.132052] ? watchdog+0x40/0x40 [ 2800.133063] ? hrtimer_interrupt+0x128/0x2c0 [ 2800.134358] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2800.135927] ? apic_timer_interrupt+0xf/0x20 [ 2800.137325] [ 2800.138118] ? native_safe_halt+0x2e/0x50 [ 2800.139339] kvm_wait+0xcd/0xe0 [ 2800.140466] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2800.142588] do_raw_spin_lock+0x177/0x190 [ 2800.144745] _raw_spin_lock+0x21/0x30 [ 2800.146280] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2800.148656] process_one_work+0x2c8/0x700 [ 2800.150846] worker_thread+0x296/0x6e0 [ 2800.151914] ? rescuer_thread+0x570/0x570 [ 2800.153191] kthread+0x1d1/0x200 [ 2800.154138] ? set_kthread_struct+0x70/0x70 [ 2800.155211] ret_from_fork+0x1f/0x30 [ 2812.054047] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2812.057718] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2812.077759] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2812.081481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2812.084795] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2812.086710] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2812.093181] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2812.095771] RAX: 0000000000000dcd RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2812.098380] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2812.100650] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2812.103787] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2812.105947] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2812.109006] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2812.112173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2812.114234] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2812.116099] Call Trace: [ 2812.116844] [ 2812.117503] ? show_regs.cold.9+0x22/0x2f [ 2812.118779] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2812.120354] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2812.121564] ? watchdog+0x40/0x40 [ 2812.122675] ? hrtimer_interrupt+0x128/0x2c0 [ 2812.123745] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2812.125048] ? apic_timer_interrupt+0xf/0x20 [ 2812.126467] [ 2812.127090] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2812.129021] do_raw_spin_lock+0x177/0x190 [ 2812.130021] _raw_spin_lock+0x21/0x30 [ 2812.131144] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2812.133698] ? do_raw_spin_unlock+0x75/0x190 [ 2812.135719] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2812.137991] ? do_wait_intr+0xf0/0xf0 [ 2812.139720] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2812.141864] mdc_close+0x53b/0xd70 [mdc] [ 2812.143541] lmv_close+0x18d/0x490 [lmv] [ 2812.145039] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2812.147078] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2812.148718] ll_md_close+0x274/0x800 [lustre] [ 2812.150160] ll_file_release+0x128/0xc50 [lustre] [ 2812.152349] __fput+0x10d/0x440 [ 2812.153858] ____fput+0x12/0x20 [ 2812.154948] task_work_run+0xc6/0x110 [ 2812.156891] exit_to_usermode_loop+0x1dd/0x1f0 [ 2812.159039] do_syscall_64+0x3d6/0x440 [ 2812.160384] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2812.162516] RIP: 0033:0x7fdc0afbfcd8 [ 2812.164036] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2812.171836] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2812.174905] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2812.177892] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2812.180458] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2812.183654] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2812.186830] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2822.624062] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 797s! [ 2822.631085] Showing busy workqueues and worker pools: [ 2822.633737] workqueue events: flags=0x0 [ 2822.635975] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2822.644458] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2822.646817] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2822.660766] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2822.673348] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2822.675710] pending: kfree_rcu_monitor [ 2822.678676] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2822.683659] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2822.691564] workqueue events_long: flags=0x0 [ 2822.693160] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2822.697932] pending: lu_objects_flush [obdclass] [ 2822.700571] workqueue events_unbound: flags=0x2 [ 2822.702588] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2822.706446] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2822.710891] workqueue events_power_efficient: flags=0x80 [ 2822.714970] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2822.719195] pending: neigh_periodic_work, check_lifetime [ 2822.721779] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2822.726523] pending: do_cache_clean, neigh_periodic_work [ 2822.731705] workqueue mm_percpu_wq: flags=0x8 [ 2822.733747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2822.736678] pending: lru_add_drain_per_cpu [ 2822.746392] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2822.753797] pending: vmstat_update, lru_add_drain_per_cpu [ 2822.758512] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2822.764623] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2822.770527] workqueue writeback: flags=0x4a [ 2822.772802] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2822.777099] pending: wb_workfn, wb_workfn [ 2822.779288] workqueue xprtiod: flags=0xa [ 2822.781973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2822.786072] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2822.790777] workqueue ipv6_addrconf: flags=0x40008 [ 2822.795022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2822.797686] pending: addrconf_verify_work [ 2822.800689] workqueue knbd0-recv: flags=0x1a [ 2822.804348] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2822.809760] in-flight: 655:recv_work [ 2822.812180] workqueue ptlrpc_pinger: flags=0x2 [ 2822.815826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2822.832080] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2822.838904] pool 8: cpus=0-3 flags=0x4 nice=0 hung=797s workers=4 idle: 36396 38 11188 [ 2822.850105] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2813s workers=2 idle: 658 [ 2824.055393] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 2824.062836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2824.085961] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2824.094282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2824.099281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2824.101607] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2824.113654] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2824.116959] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2824.121620] RDX: 0000000000007270 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2824.125743] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2824.128859] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2824.132737] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2824.135608] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2824.138618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2824.140901] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2824.143920] Call Trace: [ 2824.145042] [ 2824.145882] ? show_regs.cold.9+0x22/0x2f [ 2824.147750] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2824.149893] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2824.151659] ? watchdog+0x40/0x40 [ 2824.153128] ? hrtimer_interrupt+0x128/0x2c0 [ 2824.155089] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2824.157366] ? apic_timer_interrupt+0xf/0x20 [ 2824.159229] [ 2824.160089] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2824.162391] do_raw_spin_lock+0x177/0x190 [ 2824.164109] _raw_spin_lock+0x21/0x30 [ 2824.165375] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2824.167687] ? do_raw_spin_unlock+0x75/0x190 [ 2824.169289] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2824.171415] ? do_wait_intr+0xf0/0xf0 [ 2824.172676] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2824.174984] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2824.177487] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2824.179888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2824.182199] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2824.184467] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2824.186638] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2824.188611] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2824.190854] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2824.193515] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2824.195914] lmv_intent_open+0x338/0x1150 [lmv] [ 2824.197810] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2824.200186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2824.202484] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2824.204451] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2824.206559] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2824.208725] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2824.210847] ? do_raw_spin_unlock+0x75/0x190 [ 2824.212743] ? _raw_spin_unlock+0x12/0x30 [ 2824.214081] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2824.215761] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2824.217841] ll_atomic_open+0x36a/0xd70 [lustre] [ 2824.220486] lookup_open+0x356/0xb40 [ 2824.221968] do_last+0x313/0xfc0 [ 2824.223616] ? nd_jump_root+0xe5/0x160 [ 2824.225899] ? path_init+0x437/0x520 [ 2824.227844] path_openat+0xf7/0x500 [ 2824.229503] ? do_raw_spin_unlock+0x75/0x190 [ 2824.231541] do_filp_open+0x99/0x140 [ 2824.233036] ? getname_flags+0x6e/0x330 [ 2824.234335] ? __check_object_size+0xff/0x256 [ 2824.236512] ? do_raw_spin_unlock+0x75/0x190 [ 2824.238324] ? _raw_spin_unlock+0x12/0x30 [ 2824.240536] do_sys_openat2+0x2b4/0x410 [ 2824.242600] do_sys_open+0x73/0xa0 [ 2824.244595] __x64_sys_openat+0x24/0x30 [ 2824.246316] do_syscall_64+0xc1/0x440 [ 2824.248439] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2824.251097] RIP: 0033:0x7fdc0afbf332 [ 2824.252838] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2824.260604] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2824.263529] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2824.266404] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2824.269458] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2824.272374] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2824.275002] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2828.051146] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2828.055955] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2828.088239] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2828.099074] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2828.106592] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2828.113396] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2828.117213] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2828.128105] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2828.133524] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2828.137700] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2828.148529] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2828.155552] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2828.160496] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2828.164618] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2828.168202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2828.171286] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2828.177765] Call Trace: [ 2828.178640] [ 2828.180660] ? show_regs.cold.9+0x22/0x2f [ 2828.182100] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2828.183718] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2828.190373] ? watchdog+0x40/0x40 [ 2828.192249] ? hrtimer_interrupt+0x128/0x2c0 [ 2828.196452] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2828.197705] ? apic_timer_interrupt+0xf/0x20 [ 2828.201506] [ 2828.207019] ? native_safe_halt+0x2e/0x50 [ 2828.209187] kvm_wait+0xcd/0xe0 [ 2828.210888] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2828.212195] do_raw_spin_lock+0x177/0x190 [ 2828.213319] _raw_spin_lock+0x21/0x30 [ 2828.214384] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2828.219824] process_one_work+0x2c8/0x700 [ 2828.220964] worker_thread+0x296/0x6e0 [ 2828.221816] ? rescuer_thread+0x570/0x570 [ 2828.223837] kthread+0x1d1/0x200 [ 2828.224646] ? set_kthread_struct+0x70/0x70 [ 2828.226719] ret_from_fork+0x1f/0x30 [ 2840.057095] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 2840.070014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2840.124107] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2840.140657] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2840.152639] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2840.160462] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2840.183340] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2840.187540] RAX: 0000000000007e86 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2840.192670] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2840.200365] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2840.203032] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2840.214861] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2840.225460] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2840.232527] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2840.236470] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2840.245315] Call Trace: [ 2840.246443] [ 2840.248697] ? show_regs.cold.9+0x22/0x2f [ 2840.252305] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2840.256537] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2840.258540] ? watchdog+0x40/0x40 [ 2840.266255] ? hrtimer_interrupt+0x128/0x2c0 [ 2840.267863] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2840.273532] ? apic_timer_interrupt+0xf/0x20 [ 2840.275196] [ 2840.276000] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2840.288393] do_raw_spin_lock+0x177/0x190 [ 2840.289778] _raw_spin_lock+0x21/0x30 [ 2840.298082] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2840.300769] ? do_raw_spin_unlock+0x75/0x190 [ 2840.303203] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2840.308434] ? do_wait_intr+0xf0/0xf0 [ 2840.310467] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2840.315754] mdc_close+0x53b/0xd70 [mdc] [ 2840.317596] lmv_close+0x18d/0x490 [lmv] [ 2840.319629] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2840.325691] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2840.328179] ll_md_close+0x274/0x800 [lustre] [ 2840.333569] ll_file_release+0x128/0xc50 [lustre] [ 2840.335472] __fput+0x10d/0x440 [ 2840.338337] ____fput+0x12/0x20 [ 2840.339968] task_work_run+0xc6/0x110 [ 2840.342477] exit_to_usermode_loop+0x1dd/0x1f0 [ 2840.346255] do_syscall_64+0x3d6/0x440 [ 2840.347962] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2840.349804] RIP: 0033:0x7fdc0afbfcd8 [ 2840.353501] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2840.372011] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2840.376960] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2840.380973] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2840.392050] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2840.394703] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2840.403065] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2848.048485] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2848.050886] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2848.071695] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2848.076512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2848.080392] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2848.082637] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2848.089199] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2848.091826] RAX: 0000000000006a8f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2848.094165] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2848.096772] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2848.099323] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2848.101992] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2848.105004] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2848.107831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2848.109837] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2848.112372] Call Trace: [ 2848.113320] [ 2848.114123] ? show_regs.cold.9+0x22/0x2f [ 2848.115695] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2848.117436] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2848.119151] ? watchdog+0x40/0x40 [ 2848.120649] ? hrtimer_interrupt+0x128/0x2c0 [ 2848.122237] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2848.124071] ? apic_timer_interrupt+0xf/0x20 [ 2848.125672] [ 2848.126412] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2848.128314] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2848.130058] ? dump_stack+0xf6/0x10e [ 2848.131319] do_raw_spin_lock+0x177/0x190 [ 2848.132783] _raw_spin_lock+0x21/0x30 [ 2848.134199] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2848.136533] obd_mod_free+0x46/0x140 [ptlrpc] [ 2848.138437] mdc_commit_open+0x91/0xb0 [mdc] [ 2848.140022] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2848.142103] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2848.144244] after_reply+0xe2c/0x1780 [ptlrpc] [ 2848.146161] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2848.148214] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2848.149955] ? do_wait_intr+0xf0/0xf0 [ 2848.151097] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2848.153174] kthread+0x1d1/0x200 [ 2848.154786] ? set_kthread_struct+0x70/0x70 [ 2848.156293] ret_from_fork+0x1f/0x30 [ 2852.052021] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 2852.056309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2852.080400] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2852.086837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2852.091702] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2852.094547] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2852.104664] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2852.110219] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2852.114892] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2852.118885] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2852.122490] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2852.126199] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2852.129617] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2852.135564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2852.143372] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2852.151989] Call Trace: [ 2852.156902] [ 2852.158614] ? show_regs.cold.9+0x22/0x2f [ 2852.164887] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2852.170399] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2852.174389] ? watchdog+0x40/0x40 [ 2852.176702] ? hrtimer_interrupt+0x128/0x2c0 [ 2852.180863] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2852.184861] ? apic_timer_interrupt+0xf/0x20 [ 2852.188863] [ 2852.190942] ? native_safe_halt+0x2e/0x50 [ 2852.193723] kvm_wait+0xcd/0xe0 [ 2852.195722] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2852.199393] do_raw_spin_lock+0x177/0x190 [ 2852.201790] _raw_spin_lock+0x21/0x30 [ 2852.204026] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2852.209143] ? do_raw_spin_unlock+0x75/0x190 [ 2852.211984] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2852.216090] ? do_wait_intr+0xf0/0xf0 [ 2852.219472] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2852.227489] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2852.235980] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2852.246673] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2852.252478] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2852.255898] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2852.262895] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2852.268658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2852.274132] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2852.276084] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2852.281181] lmv_intent_open+0x338/0x1150 [lmv] [ 2852.287979] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2852.292837] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2852.295891] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2852.300618] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2852.304889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2852.308024] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2852.309731] ? do_raw_spin_unlock+0x75/0x190 [ 2852.311673] ? _raw_spin_unlock+0x12/0x30 [ 2852.313704] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2852.315439] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2852.317651] ll_atomic_open+0x36a/0xd70 [lustre] [ 2852.321310] lookup_open+0x356/0xb40 [ 2852.326979] do_last+0x313/0xfc0 [ 2852.329756] ? nd_jump_root+0xe5/0x160 [ 2852.331924] ? path_init+0x437/0x520 [ 2852.334055] path_openat+0xf7/0x500 [ 2852.339040] ? do_raw_spin_unlock+0x75/0x190 [ 2852.341457] do_filp_open+0x99/0x140 [ 2852.344079] ? getname_flags+0x6e/0x330 [ 2852.347397] ? __check_object_size+0xff/0x256 [ 2852.353010] ? do_raw_spin_unlock+0x75/0x190 [ 2852.354474] ? _raw_spin_unlock+0x12/0x30 [ 2852.356440] do_sys_openat2+0x2b4/0x410 [ 2852.358190] do_sys_open+0x73/0xa0 [ 2852.362403] __x64_sys_openat+0x24/0x30 [ 2852.366512] do_syscall_64+0xc1/0x440 [ 2852.368096] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2852.374031] RIP: 0033:0x7fdc0afbf332 [ 2852.376678] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2852.393870] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2852.402848] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2852.408372] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2852.416430] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2852.423179] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2852.430289] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2853.346319] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 828s! [ 2853.350709] Showing busy workqueues and worker pools: [ 2853.355818] workqueue events: flags=0x0 [ 2853.367765] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2853.380871] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2853.391568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2853.393215] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2853.406568] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2853.411599] pending: kfree_rcu_monitor [ 2853.414604] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2853.418421] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2853.424815] workqueue events_long: flags=0x0 [ 2853.427876] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2853.431134] pending: lu_objects_flush [obdclass] [ 2853.433072] workqueue events_unbound: flags=0x2 [ 2853.434717] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2853.436779] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2853.442358] workqueue events_power_efficient: flags=0x80 [ 2853.444525] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2853.446807] pending: neigh_periodic_work, check_lifetime [ 2853.449626] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2853.453060] pending: do_cache_clean, neigh_periodic_work [ 2853.456144] workqueue mm_percpu_wq: flags=0x8 [ 2853.458057] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2853.461388] pending: lru_add_drain_per_cpu [ 2853.463659] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2853.466988] pending: vmstat_update, lru_add_drain_per_cpu [ 2853.470032] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2853.473819] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2853.477089] workqueue writeback: flags=0x4a [ 2853.479202] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2853.482323] pending: wb_workfn, wb_workfn [ 2853.483962] workqueue xprtiod: flags=0xa [ 2853.486204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2853.490121] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2853.493384] workqueue ipv6_addrconf: flags=0x40008 [ 2853.496575] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2853.500065] pending: addrconf_verify_work [ 2853.501671] workqueue knbd0-recv: flags=0x1a [ 2853.503689] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2853.506241] in-flight: 655:recv_work [ 2853.509247] workqueue ptlrpc_pinger: flags=0x2 [ 2853.512741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2853.516396] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2853.520101] pool 8: cpus=0-3 flags=0x4 nice=0 hung=828s workers=4 idle: 36396 38 11188 [ 2853.526073] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2844s workers=2 idle: 658 [ 2856.051073] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2856.056296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2856.087025] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2856.095513] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2856.104528] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2856.109669] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2856.112243] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2856.126213] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2856.130985] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2856.139244] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2856.145870] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2856.152339] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2856.156878] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2856.161283] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2856.177628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2856.180038] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2856.186266] Call Trace: [ 2856.187539] [ 2856.191671] ? show_regs.cold.9+0x22/0x2f [ 2856.194201] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2856.200072] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2856.202284] ? watchdog+0x40/0x40 [ 2856.203264] ? hrtimer_interrupt+0x128/0x2c0 [ 2856.204322] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2856.211231] ? apic_timer_interrupt+0xf/0x20 [ 2856.222800] [ 2856.225494] ? native_safe_halt+0x2e/0x50 [ 2856.227338] kvm_wait+0xcd/0xe0 [ 2856.233952] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2856.237469] do_raw_spin_lock+0x177/0x190 [ 2856.239518] _raw_spin_lock+0x21/0x30 [ 2856.244222] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2856.247877] process_one_work+0x2c8/0x700 [ 2856.254723] worker_thread+0x296/0x6e0 [ 2856.260883] ? rescuer_thread+0x570/0x570 [ 2856.263504] kthread+0x1d1/0x200 [ 2856.264709] ? set_kthread_struct+0x70/0x70 [ 2856.267320] ret_from_fork+0x1f/0x30 [ 2868.054027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 2868.056788] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2868.123487] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2868.138443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2868.146546] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2868.150957] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2868.171980] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2868.183200] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2868.196693] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2868.203115] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2868.212455] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2868.217083] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2868.220586] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2868.225191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2868.227919] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2868.231360] Call Trace: [ 2868.232590] [ 2868.234416] ? show_regs.cold.9+0x22/0x2f [ 2868.236868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2868.241444] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2868.245844] ? watchdog+0x40/0x40 [ 2868.247527] ? hrtimer_interrupt+0x128/0x2c0 [ 2868.248993] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2868.250590] ? apic_timer_interrupt+0xf/0x20 [ 2868.257445] [ 2868.259536] ? native_safe_halt+0x2e/0x50 [ 2868.263944] kvm_wait+0xcd/0xe0 [ 2868.267500] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2868.271868] do_raw_spin_lock+0x177/0x190 [ 2868.273325] _raw_spin_lock+0x21/0x30 [ 2868.276001] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2868.278652] ? do_raw_spin_unlock+0x75/0x190 [ 2868.282648] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2868.286433] ? do_wait_intr+0xf0/0xf0 [ 2868.288402] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2868.293971] mdc_close+0x53b/0xd70 [mdc] [ 2868.297512] lmv_close+0x18d/0x490 [lmv] [ 2868.299481] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2868.304946] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2868.308714] ll_md_close+0x274/0x800 [lustre] [ 2868.312902] ll_file_release+0x128/0xc50 [lustre] [ 2868.315029] __fput+0x10d/0x440 [ 2868.316147] ____fput+0x12/0x20 [ 2868.317858] task_work_run+0xc6/0x110 [ 2868.319604] exit_to_usermode_loop+0x1dd/0x1f0 [ 2868.321710] do_syscall_64+0x3d6/0x440 [ 2868.328480] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2868.331960] RIP: 0033:0x7fdc0afbfcd8 [ 2868.334325] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2868.345231] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2868.349179] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2868.353000] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2868.356259] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2868.360002] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2868.363686] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2880.048208] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 2880.052337] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 2880.054947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2880.055041] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2880.055050] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2880.055052] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2880.055067] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2880.055072] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2880.055076] RAX: 00000000000070fb RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2880.055079] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2880.055081] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2880.055083] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2880.055085] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2880.055088] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2880.055091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2880.055094] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2880.055104] Call Trace: [ 2880.055111] [ 2880.055117] ? show_regs.cold.9+0x22/0x2f [ 2880.055127] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2880.055134] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2880.055139] ? watchdog+0x40/0x40 [ 2880.055143] ? hrtimer_interrupt+0x128/0x2c0 [ 2880.055159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2880.055167] ? apic_timer_interrupt+0xf/0x20 [ 2880.055172] [ 2880.055175] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2880.055180] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2880.055184] ? dump_stack+0xf6/0x10e [ 2880.055190] do_raw_spin_lock+0x177/0x190 [ 2880.055195] _raw_spin_lock+0x21/0x30 [ 2880.055201] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2880.055683] obd_mod_free+0x46/0x140 [ptlrpc] [ 2880.055885] mdc_commit_open+0x91/0xb0 [mdc] [ 2880.055921] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2880.056291] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2880.056572] after_reply+0xe2c/0x1780 [ptlrpc] [ 2880.056771] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2880.057146] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2880.067072] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [ 2880.115700] ? do_wait_intr+0xf0/0xf0 [ 2880.115726] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2880.116157] kthread+0x1d1/0x200 [ 2880.116163] ? set_kthread_struct+0x70/0x70 [ 2880.116167] ret_from_fork+0x1f/0x30 [ 2880.255070] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2880.268568] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2880.273595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2880.277662] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2880.280121] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2880.287680] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2880.290741] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2880.294985] RDX: 0000000000003e4d RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2880.298110] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2880.300793] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2880.304649] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2880.307836] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2880.311197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2880.314707] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2880.318590] Call Trace: [ 2880.319662] [ 2880.320569] ? show_regs.cold.9+0x22/0x2f [ 2880.322198] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2880.324341] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2880.325998] ? watchdog+0x40/0x40 [ 2880.327725] ? hrtimer_interrupt+0x128/0x2c0 [ 2880.329772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2880.332648] ? apic_timer_interrupt+0xf/0x20 [ 2880.334554] [ 2880.335780] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2880.338487] do_raw_spin_lock+0x177/0x190 [ 2880.340128] _raw_spin_lock+0x21/0x30 [ 2880.341856] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2880.344639] ? do_raw_spin_unlock+0x75/0x190 [ 2880.346634] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2880.349217] ? do_wait_intr+0xf0/0xf0 [ 2880.351578] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2880.354733] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2880.357705] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2880.361596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2880.363881] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2880.366289] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2880.370223] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2880.372616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2880.375843] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2880.378571] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2880.381078] lmv_intent_open+0x338/0x1150 [lmv] [ 2880.382724] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2880.385339] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2880.387799] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2880.389980] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2880.392335] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2880.395814] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2880.398516] ? do_raw_spin_unlock+0x75/0x190 [ 2880.400396] ? _raw_spin_unlock+0x12/0x30 [ 2880.402305] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2880.405638] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2880.408897] ll_atomic_open+0x36a/0xd70 [lustre] [ 2880.412597] lookup_open+0x356/0xb40 [ 2880.414226] do_last+0x313/0xfc0 [ 2880.415769] ? nd_jump_root+0xe5/0x160 [ 2880.417441] ? path_init+0x437/0x520 [ 2880.419149] path_openat+0xf7/0x500 [ 2880.420765] ? do_raw_spin_unlock+0x75/0x190 [ 2880.422673] do_filp_open+0x99/0x140 [ 2880.424058] ? getname_flags+0x6e/0x330 [ 2880.426173] ? __check_object_size+0xff/0x256 [ 2880.427856] ? do_raw_spin_unlock+0x75/0x190 [ 2880.430203] ? _raw_spin_unlock+0x12/0x30 [ 2880.432067] do_sys_openat2+0x2b4/0x410 [ 2880.433764] do_sys_open+0x73/0xa0 [ 2880.435298] __x64_sys_openat+0x24/0x30 [ 2880.436941] do_syscall_64+0xc1/0x440 [ 2880.438596] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2880.441826] RIP: 0033:0x7fdc0afbf332 [ 2880.443504] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2880.451209] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2880.453847] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2880.456326] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2880.461348] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2880.464385] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2880.467619] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2883.556196] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 858s! [ 2883.567200] Showing busy workqueues and worker pools: [ 2883.578631] workqueue events: flags=0x0 [ 2883.582043] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2883.591895] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2883.602433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2883.607812] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2883.619464] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2883.624916] pending: kfree_rcu_monitor [ 2883.629446] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2883.634494] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2883.643794] workqueue events_long: flags=0x0 [ 2883.650351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2883.655093] pending: lu_objects_flush [obdclass] [ 2883.659178] workqueue events_unbound: flags=0x2 [ 2883.661764] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2883.665881] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2883.676789] workqueue events_power_efficient: flags=0x80 [ 2883.679857] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2883.685824] pending: neigh_periodic_work, check_lifetime [ 2883.690235] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2883.693837] pending: do_cache_clean, neigh_periodic_work [ 2883.697045] workqueue mm_percpu_wq: flags=0x8 [ 2883.699931] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2883.706667] pending: lru_add_drain_per_cpu [ 2883.711063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2883.716764] pending: vmstat_update, lru_add_drain_per_cpu [ 2883.718775] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2883.721831] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2883.725803] workqueue writeback: flags=0x4a [ 2883.727098] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2883.730102] pending: wb_workfn, wb_workfn [ 2883.735807] workqueue xprtiod: flags=0xa [ 2883.738471] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2883.744060] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2883.747815] workqueue ipv6_addrconf: flags=0x40008 [ 2883.752434] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2883.757172] pending: addrconf_verify_work [ 2883.759747] workqueue knbd0-recv: flags=0x1a [ 2883.764640] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2883.769858] in-flight: 655:recv_work [ 2883.771099] workqueue ptlrpc_pinger: flags=0x2 [ 2883.775895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2883.782612] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2883.787482] pool 8: cpus=0-3 flags=0x4 nice=0 hung=858s workers=4 idle: 36396 38 11188 [ 2883.795677] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2874s workers=2 idle: 658 [ 2884.054096] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 2884.059755] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2884.102825] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2884.118654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2884.130255] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2884.136871] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2884.143222] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2884.160834] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2884.170329] RAX: 0000000000007f89 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2884.180289] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2884.195205] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2884.207537] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2884.215957] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2884.225186] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2884.227964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2884.236518] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2884.243421] Call Trace: [ 2884.247304] [ 2884.250721] ? show_regs.cold.9+0x22/0x2f [ 2884.258556] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2884.265887] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2884.268588] ? watchdog+0x40/0x40 [ 2884.269856] ? hrtimer_interrupt+0x128/0x2c0 [ 2884.271077] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2884.276679] ? apic_timer_interrupt+0xf/0x20 [ 2884.278086] [ 2884.280164] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2884.282731] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2884.288677] do_raw_spin_lock+0x177/0x190 [ 2884.290857] _raw_spin_lock+0x21/0x30 [ 2884.292807] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2884.299740] process_one_work+0x2c8/0x700 [ 2884.304819] worker_thread+0x296/0x6e0 [ 2884.306756] ? rescuer_thread+0x570/0x570 [ 2884.309335] kthread+0x1d1/0x200 [ 2884.310869] ? set_kthread_struct+0x70/0x70 [ 2884.314235] ret_from_fork+0x1f/0x30 [ 2896.054109] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 2896.064430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2896.084641] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2896.089302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2896.092718] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 2896.095311] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 2896.102900] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2896.105866] RAX: 0000000000002c51 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2896.108024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2896.110913] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2896.113507] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2896.115895] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2896.118961] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2896.121186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2896.124693] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2896.126823] Call Trace: [ 2896.127641] [ 2896.128803] ? show_regs.cold.9+0x22/0x2f [ 2896.130029] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2896.132071] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2896.133738] ? watchdog+0x40/0x40 [ 2896.135010] ? hrtimer_interrupt+0x128/0x2c0 [ 2896.136513] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2896.138329] ? apic_timer_interrupt+0xf/0x20 [ 2896.139805] [ 2896.140582] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 2896.142552] do_raw_spin_lock+0x177/0x190 [ 2896.143883] _raw_spin_lock+0x21/0x30 [ 2896.145040] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2896.147163] ? do_raw_spin_unlock+0x75/0x190 [ 2896.148833] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2896.150722] ? do_wait_intr+0xf0/0xf0 [ 2896.151825] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2896.154102] mdc_close+0x53b/0xd70 [mdc] [ 2896.155582] lmv_close+0x18d/0x490 [lmv] [ 2896.157260] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2896.159122] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2896.161080] ll_md_close+0x274/0x800 [lustre] [ 2896.162699] ll_file_release+0x128/0xc50 [lustre] [ 2896.164795] __fput+0x10d/0x440 [ 2896.165991] ____fput+0x12/0x20 [ 2896.166990] task_work_run+0xc6/0x110 [ 2896.168370] exit_to_usermode_loop+0x1dd/0x1f0 [ 2896.170161] do_syscall_64+0x3d6/0x440 [ 2896.171267] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2896.173344] RIP: 0033:0x7fdc0afbfcd8 [ 2896.174699] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2896.182023] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2896.184926] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2896.187776] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2896.190221] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2896.193052] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2896.195610] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2908.048041] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 2908.052812] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2908.054147] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 2908.099856] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2908.099873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2908.105101] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2908.105803] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2908.105811] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2908.105814] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2908.105832] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2908.105836] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2908.105840] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2908.105842] RDX: 0000000000003101 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2908.105844] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2908.105846] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2908.105848] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2908.105851] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2908.105854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2908.105856] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2908.105864] Call Trace: [ 2908.105869] [ 2908.105877] ? show_regs.cold.9+0x22/0x2f [ 2908.105886] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2908.105892] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2908.105897] ? watchdog+0x40/0x40 [ 2908.105901] ? hrtimer_interrupt+0x128/0x2c0 [ 2908.105905] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2908.105914] ? apic_timer_interrupt+0xf/0x20 [ 2908.105919] [ 2908.105922] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2908.105939] do_raw_spin_lock+0x177/0x190 [ 2908.105945] _raw_spin_lock+0x21/0x30 [ 2908.105951] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2908.107046] ? do_raw_spin_unlock+0x75/0x190 [ 2908.107061] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2908.107327] ? do_wait_intr+0xf0/0xf0 [ 2908.107336] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2908.107826] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2908.108081] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2908.108392] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2908.108732] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2908.108779] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2908.108808] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2908.108988] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2908.120173] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2908.120196] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2908.120201] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2908.120207] RAX: 0000000000007a34 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2908.120210] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2908.120212] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2908.120215] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2908.120218] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2908.120221] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2908.120225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2908.120227] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2908.120476] Call Trace: [ 2908.120482] [ 2908.120488] ? show_regs.cold.9+0x22/0x2f [ 2908.120498] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2908.120504] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2908.120509] ? watchdog+0x40/0x40 [ 2908.120514] ? hrtimer_interrupt+0x128/0x2c0 [ 2908.120577] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2908.120590] ? apic_timer_interrupt+0xf/0x20 [ 2908.120595] [ 2908.120598] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2908.120603] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2908.120607] ? dump_stack+0xf6/0x10e [ 2908.120612] do_raw_spin_lock+0x177/0x190 [ 2908.120618] _raw_spin_lock+0x21/0x30 [ 2908.120625] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2908.126883] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2908.127447] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2908.127488] lmv_intent_open+0x338/0x1150 [lmv] [ 2908.127838] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2908.127975] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2908.128058] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2908.128083] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2908.173808] obd_mod_free+0x46/0x140 [ptlrpc] [ 2908.174199] mdc_commit_open+0x91/0xb0 [mdc] [ 2908.174242] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2908.174476] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2908.174679] after_reply+0xe2c/0x1780 [ptlrpc] [ 2908.174875] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2908.175189] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2908.175579] ? do_wait_intr+0xf0/0xf0 [ 2908.175593] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2908.175877] kthread+0x1d1/0x200 [ 2908.175884] ? set_kthread_struct+0x70/0x70 [ 2908.175888] ret_from_fork+0x1f/0x30 [ 2908.537794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2908.540108] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2908.542175] ? do_raw_spin_unlock+0x75/0x190 [ 2908.543941] ? _raw_spin_unlock+0x12/0x30 [ 2908.545616] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2908.547719] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2908.549802] ll_atomic_open+0x36a/0xd70 [lustre] [ 2908.551876] lookup_open+0x356/0xb40 [ 2908.553234] do_last+0x313/0xfc0 [ 2908.554791] ? nd_jump_root+0xe5/0x160 [ 2908.556244] ? path_init+0x437/0x520 [ 2908.557815] path_openat+0xf7/0x500 [ 2908.559092] ? do_raw_spin_unlock+0x75/0x190 [ 2908.560735] do_filp_open+0x99/0x140 [ 2908.562029] ? getname_flags+0x6e/0x330 [ 2908.563365] ? __check_object_size+0xff/0x256 [ 2908.565394] ? do_raw_spin_unlock+0x75/0x190 [ 2908.567209] ? _raw_spin_unlock+0x12/0x30 [ 2908.569214] do_sys_openat2+0x2b4/0x410 [ 2908.570831] do_sys_open+0x73/0xa0 [ 2908.572252] __x64_sys_openat+0x24/0x30 [ 2908.573742] do_syscall_64+0xc1/0x440 [ 2908.575181] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2908.577831] RIP: 0033:0x7fdc0afbf332 [ 2908.579384] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2908.589676] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2908.592433] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2908.595609] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2908.598221] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2908.601078] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2908.604096] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2912.051046] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 2912.054073] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2912.110488] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2912.119016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2912.126046] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2912.139232] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2912.146111] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2912.159440] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2912.179189] RAX: 0000000000006d86 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2912.199135] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2912.212011] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2912.219829] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2912.232917] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2912.247481] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2912.261931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2912.276234] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2912.293729] Call Trace: [ 2912.295633] [ 2912.301651] ? show_regs.cold.9+0x22/0x2f [ 2912.309802] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2912.321913] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2912.329179] ? watchdog+0x40/0x40 [ 2912.339598] ? hrtimer_interrupt+0x128/0x2c0 [ 2912.345697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2912.362683] ? apic_timer_interrupt+0xf/0x20 [ 2912.368707] [ 2912.372721] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2912.383384] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2912.394014] do_raw_spin_lock+0x177/0x190 [ 2912.403828] _raw_spin_lock+0x21/0x30 [ 2912.416270] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2912.418438] process_one_work+0x2c8/0x700 [ 2912.425007] worker_thread+0x296/0x6e0 [ 2912.428507] ? rescuer_thread+0x570/0x570 [ 2912.431917] kthread+0x1d1/0x200 [ 2912.433207] ? set_kthread_struct+0x70/0x70 [ 2912.435912] ret_from_fork+0x1f/0x30 [ 2914.281564] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 889s! [ 2914.288407] Showing busy workqueues and worker pools: [ 2914.291755] workqueue events: flags=0x0 [ 2914.292955] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2914.294681] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2914.302486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2914.304799] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2914.317037] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2914.327926] pending: kfree_rcu_monitor [ 2914.329169] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2914.330788] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2914.339732] workqueue events_long: flags=0x0 [ 2914.342064] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2914.344833] pending: lu_objects_flush [obdclass] [ 2914.349018] workqueue events_unbound: flags=0x2 [ 2914.350961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2914.353738] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2914.358718] workqueue events_power_efficient: flags=0x80 [ 2914.362074] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2914.365691] pending: neigh_periodic_work, check_lifetime [ 2914.368042] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2914.370909] pending: do_cache_clean, neigh_periodic_work [ 2914.373232] workqueue mm_percpu_wq: flags=0x8 [ 2914.384985] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2914.387379] pending: lru_add_drain_per_cpu [ 2914.392008] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2914.395830] pending: vmstat_update, lru_add_drain_per_cpu [ 2914.397862] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2914.400726] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2914.409365] workqueue writeback: flags=0x4a [ 2914.411383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2914.414597] pending: wb_workfn, wb_workfn [ 2914.416843] workqueue xprtiod: flags=0xa [ 2914.426476] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2914.428133] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2914.431007] workqueue ipv6_addrconf: flags=0x40008 [ 2914.437628] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2914.440857] pending: addrconf_verify_work [ 2914.448906] workqueue knbd0-recv: flags=0x1a [ 2914.450957] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2914.456775] in-flight: 655:recv_work [ 2914.460708] workqueue ptlrpc_pinger: flags=0x2 [ 2914.465152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2914.469408] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2914.473894] pool 8: cpus=0-3 flags=0x4 nice=0 hung=889s workers=4 idle: 36396 38 11188 [ 2914.482769] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2905s workers=2 idle: 658 [ 2924.054024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 2924.058080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2924.081538] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2924.086945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2924.092134] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2924.094692] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2924.105089] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2924.110118] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2924.116754] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2924.120674] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2924.128556] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2924.132372] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2924.136606] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2924.141286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2924.144561] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2924.149610] Call Trace: [ 2924.150786] [ 2924.152403] ? show_regs.cold.9+0x22/0x2f [ 2924.154576] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2924.157809] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2924.159809] ? watchdog+0x40/0x40 [ 2924.163457] ? hrtimer_interrupt+0x128/0x2c0 [ 2924.166612] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2924.168804] ? apic_timer_interrupt+0xf/0x20 [ 2924.171113] [ 2924.172155] ? native_safe_halt+0x2e/0x50 [ 2924.174006] kvm_wait+0xcd/0xe0 [ 2924.176811] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2924.181238] do_raw_spin_lock+0x177/0x190 [ 2924.183002] _raw_spin_lock+0x21/0x30 [ 2924.185414] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2924.187525] ? do_raw_spin_unlock+0x75/0x190 [ 2924.190162] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2924.193321] ? do_wait_intr+0xf0/0xf0 [ 2924.194989] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2924.198216] mdc_close+0x53b/0xd70 [mdc] [ 2924.200752] lmv_close+0x18d/0x490 [lmv] [ 2924.203164] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2924.207724] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2924.212133] ll_md_close+0x274/0x800 [lustre] [ 2924.214310] ll_file_release+0x128/0xc50 [lustre] [ 2924.217216] __fput+0x10d/0x440 [ 2924.219341] ____fput+0x12/0x20 [ 2924.223310] task_work_run+0xc6/0x110 [ 2924.225685] exit_to_usermode_loop+0x1dd/0x1f0 [ 2924.227693] do_syscall_64+0x3d6/0x440 [ 2924.229405] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2924.233053] RIP: 0033:0x7fdc0afbfcd8 [ 2924.234632] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2924.244054] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2924.248756] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2924.252659] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2924.256516] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2924.260172] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2924.265756] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2936.060859] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2936.065592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2936.127237] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2936.138624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2936.155605] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2936.160010] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2936.188175] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2936.199661] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2936.207886] RDX: 0000000000002878 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2936.222864] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2936.236102] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2936.247867] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2936.254795] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2936.263692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2936.281040] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2936.288921] Call Trace: [ 2936.290846] [ 2936.291884] ? show_regs.cold.9+0x22/0x2f [ 2936.293754] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2936.296045] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2936.301525] ? watchdog+0x40/0x40 [ 2936.308069] ? hrtimer_interrupt+0x128/0x2c0 [ 2936.318710] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2936.328283] ? apic_timer_interrupt+0xf/0x20 [ 2936.338750] [ 2936.343499] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2936.354629] do_raw_spin_lock+0x177/0x190 [ 2936.356731] _raw_spin_lock+0x21/0x30 [ 2936.360885] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2936.367298] ? do_raw_spin_unlock+0x75/0x190 [ 2936.373465] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2936.378175] ? do_wait_intr+0xf0/0xf0 [ 2936.381172] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2936.382725] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2936.384428] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2936.402509] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2936.409796] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2936.422474] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2936.427747] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2936.429840] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2936.437293] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2936.440409] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2936.444601] lmv_intent_open+0x338/0x1150 [lmv] [ 2936.446195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2936.452662] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2936.457981] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2936.459385] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2936.463925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2936.467383] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2936.474927] ? do_raw_spin_unlock+0x75/0x190 [ 2936.477594] ? _raw_spin_unlock+0x12/0x30 [ 2936.484012] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2936.488335] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2936.495623] ll_atomic_open+0x36a/0xd70 [lustre] [ 2936.502490] lookup_open+0x356/0xb40 [ 2936.506719] do_last+0x313/0xfc0 [ 2936.510612] ? nd_jump_root+0xe5/0x160 [ 2936.515281] ? path_init+0x437/0x520 [ 2936.521266] path_openat+0xf7/0x500 [ 2936.525135] ? do_raw_spin_unlock+0x75/0x190 [ 2936.526941] do_filp_open+0x99/0x140 [ 2936.530259] ? getname_flags+0x6e/0x330 [ 2936.534335] ? __check_object_size+0xff/0x256 [ 2936.537022] ? do_raw_spin_unlock+0x75/0x190 [ 2936.540369] ? _raw_spin_unlock+0x12/0x30 [ 2936.542647] do_sys_openat2+0x2b4/0x410 [ 2936.544041] do_sys_open+0x73/0xa0 [ 2936.548348] __x64_sys_openat+0x24/0x30 [ 2936.552000] do_syscall_64+0xc1/0x440 [ 2936.553748] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2936.555343] RIP: 0033:0x7fdc0afbf332 [ 2936.557416] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2936.575402] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2936.584680] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2936.593489] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2936.602557] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2936.611515] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2936.622973] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2940.050108] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 2940.055215] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 2940.062300] Modules linked in: [ 2940.073321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2940.073645] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2940.073651] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2940.073749] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2940.075251] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2940.075316] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2940.075322] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2940.075325] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2940.075356] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2940.075361] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2940.075365] RAX: 0000000000007c8a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2940.075376] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2940.075378] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2940.075380] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2940.075381] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2940.075384] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2940.075387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2940.075389] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2940.075396] Call Trace: [ 2940.075413] [ 2940.075418] ? show_regs.cold.9+0x22/0x2f [ 2940.075444] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2940.075450] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2940.075454] ? watchdog+0x40/0x40 [ 2940.075458] ? hrtimer_interrupt+0x128/0x2c0 [ 2940.075462] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2940.075470] ? apic_timer_interrupt+0xf/0x20 [ 2940.075475] [ 2940.075478] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2940.075483] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2940.075487] ? dump_stack+0xf6/0x10e [ 2940.075493] do_raw_spin_lock+0x177/0x190 [ 2940.075497] _raw_spin_lock+0x21/0x30 [ 2940.075504] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2940.075783] obd_mod_free+0x46/0x140 [ptlrpc] [ 2940.076174] mdc_commit_open+0x91/0xb0 [mdc] [ 2940.076225] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2940.076437] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2940.077054] after_reply+0xe2c/0x1780 [ptlrpc] [ 2940.077252] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2940.077457] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2940.080221] ? do_wait_intr+0xf0/0xf0 [ 2940.080232] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2940.080457] kthread+0x1d1/0x200 [ 2940.080463] ? set_kthread_struct+0x70/0x70 [ 2940.080467] ret_from_fork+0x1f/0x30 [ 2940.277775] [ 2940.278733] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2940.284421] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2940.299562] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2940.311079] RAX: 0000000000007c18 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2940.322393] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2940.339773] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2940.348141] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2940.353793] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2940.355769] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2940.365502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2940.368758] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2940.372948] Call Trace: [ 2940.377745] [ 2940.378528] ? show_regs.cold.9+0x22/0x2f [ 2940.380626] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2940.386850] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2940.390322] ? watchdog+0x40/0x40 [ 2940.395694] ? hrtimer_interrupt+0x128/0x2c0 [ 2940.398742] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2940.406811] ? apic_timer_interrupt+0xf/0x20 [ 2940.413552] [ 2940.414199] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2940.415796] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2940.419746] do_raw_spin_lock+0x177/0x190 [ 2940.423552] _raw_spin_lock+0x21/0x30 [ 2940.428736] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2940.432782] process_one_work+0x2c8/0x700 [ 2940.437884] worker_thread+0x296/0x6e0 [ 2940.442806] ? rescuer_thread+0x570/0x570 [ 2940.448866] kthread+0x1d1/0x200 [ 2940.454683] ? set_kthread_struct+0x70/0x70 [ 2940.457737] ret_from_fork+0x1f/0x30 [ 2945.008919] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 920s! [ 2945.025859] Showing busy workqueues and worker pools: [ 2945.032130] workqueue events: flags=0x0 [ 2945.035639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2945.044236] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2945.051925] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2945.066517] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2945.076109] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2945.078902] pending: kfree_rcu_monitor [ 2945.080326] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2945.083470] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2945.095776] workqueue events_long: flags=0x0 [ 2945.097315] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2945.100933] pending: lu_objects_flush [obdclass] [ 2945.102477] workqueue events_unbound: flags=0x2 [ 2945.103954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2945.106436] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2945.110258] workqueue events_power_efficient: flags=0x80 [ 2945.112467] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2945.115273] pending: neigh_periodic_work, check_lifetime [ 2945.117653] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2945.128693] pending: do_cache_clean, neigh_periodic_work [ 2945.145137] workqueue mm_percpu_wq: flags=0x8 [ 2945.152152] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2945.166955] pending: lru_add_drain_per_cpu [ 2945.169807] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2945.173615] pending: vmstat_update, lru_add_drain_per_cpu [ 2945.181146] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2945.186939] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2945.190714] workqueue writeback: flags=0x4a [ 2945.193142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2945.196430] pending: wb_workfn, wb_workfn [ 2945.199986] workqueue xprtiod: flags=0xa [ 2945.201715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2945.204412] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2945.208811] workqueue ipv6_addrconf: flags=0x40008 [ 2945.211129] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2945.214511] pending: addrconf_verify_work [ 2945.216825] workqueue knbd0-recv: flags=0x1a [ 2945.218858] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2945.222226] in-flight: 655:recv_work [ 2945.224453] workqueue ptlrpc_pinger: flags=0x2 [ 2945.227757] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2945.232659] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2945.236367] pool 8: cpus=0-3 flags=0x4 nice=0 hung=920s workers=4 idle: 36396 38 11188 [ 2945.239361] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2935s workers=2 idle: 658 [ 2952.054524] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 2952.060901] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2952.106864] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2952.126458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2952.132747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2952.137445] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2952.162474] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2952.171654] RAX: 000000000000621a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2952.176470] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2952.181687] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2952.192503] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2952.202554] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2952.210570] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2952.222290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2952.226704] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2952.238859] Call Trace: [ 2952.242337] [ 2952.245140] ? show_regs.cold.9+0x22/0x2f [ 2952.246539] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2952.262357] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2952.264579] ? watchdog+0x40/0x40 [ 2952.271649] ? hrtimer_interrupt+0x128/0x2c0 [ 2952.275554] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2952.282456] ? apic_timer_interrupt+0xf/0x20 [ 2952.290982] [ 2952.292313] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2952.298343] do_raw_spin_lock+0x177/0x190 [ 2952.301894] _raw_spin_lock+0x21/0x30 [ 2952.303601] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2952.312720] ? do_raw_spin_unlock+0x75/0x190 [ 2952.320924] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2952.327554] ? do_wait_intr+0xf0/0xf0 [ 2952.329286] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2952.334312] mdc_close+0x53b/0xd70 [mdc] [ 2952.337121] lmv_close+0x18d/0x490 [lmv] [ 2952.338741] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2952.345963] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2952.353574] ll_md_close+0x274/0x800 [lustre] [ 2952.358399] ll_file_release+0x128/0xc50 [lustre] [ 2952.364240] __fput+0x10d/0x440 [ 2952.367436] ____fput+0x12/0x20 [ 2952.373844] task_work_run+0xc6/0x110 [ 2952.376730] exit_to_usermode_loop+0x1dd/0x1f0 [ 2952.382383] do_syscall_64+0x3d6/0x440 [ 2952.385311] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2952.393585] RIP: 0033:0x7fdc0afbfcd8 [ 2952.396469] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2952.413067] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2952.426695] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2952.430504] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2952.437552] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2952.442815] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2952.451428] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2964.052336] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 2964.057499] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2964.096523] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2964.103154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2964.107819] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2964.113841] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2964.127134] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2964.130857] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2964.135753] RDX: 000000000000376a RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2964.141209] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2964.151470] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2964.155153] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2964.160589] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2964.164474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2964.166837] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2964.182942] Call Trace: [ 2964.184455] [ 2964.186026] ? show_regs.cold.9+0x22/0x2f [ 2964.187628] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2964.191240] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2964.197135] ? watchdog+0x40/0x40 [ 2964.199090] ? hrtimer_interrupt+0x128/0x2c0 [ 2964.203801] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2964.205861] ? apic_timer_interrupt+0xf/0x20 [ 2964.210180] [ 2964.211061] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2964.214448] do_raw_spin_lock+0x177/0x190 [ 2964.216029] _raw_spin_lock+0x21/0x30 [ 2964.221890] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2964.224859] ? do_raw_spin_unlock+0x75/0x190 [ 2964.225974] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2964.228951] ? do_wait_intr+0xf0/0xf0 [ 2964.238406] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2964.242079] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2964.252644] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2964.254742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2964.263168] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2964.266032] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2964.271231] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2964.276078] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2964.280386] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2964.284058] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2964.287272] lmv_intent_open+0x338/0x1150 [lmv] [ 2964.289924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2964.294821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2964.299413] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2964.303049] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2964.306031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2964.308523] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2964.310845] ? do_raw_spin_unlock+0x75/0x190 [ 2964.312882] ? _raw_spin_unlock+0x12/0x30 [ 2964.314778] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2964.317110] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2964.319456] ll_atomic_open+0x36a/0xd70 [lustre] [ 2964.321820] lookup_open+0x356/0xb40 [ 2964.323429] do_last+0x313/0xfc0 [ 2964.324977] ? nd_jump_root+0xe5/0x160 [ 2964.326739] ? path_init+0x437/0x520 [ 2964.328401] path_openat+0xf7/0x500 [ 2964.330038] ? do_raw_spin_unlock+0x75/0x190 [ 2964.332021] do_filp_open+0x99/0x140 [ 2964.333663] ? getname_flags+0x6e/0x330 [ 2964.335937] ? __check_object_size+0xff/0x256 [ 2964.338655] ? do_raw_spin_unlock+0x75/0x190 [ 2964.341436] ? _raw_spin_unlock+0x12/0x30 [ 2964.343475] do_sys_openat2+0x2b4/0x410 [ 2964.349153] do_sys_open+0x73/0xa0 [ 2964.350456] __x64_sys_openat+0x24/0x30 [ 2964.351860] do_syscall_64+0xc1/0x440 [ 2964.354233] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2964.357782] RIP: 0033:0x7fdc0afbf332 [ 2964.359415] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2964.367936] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2964.374714] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2964.378963] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2964.383161] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2964.388107] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2964.392484] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2968.051020] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 2968.053979] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2968.068439] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2968.072711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2968.076175] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2968.078639] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2968.080077] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2968.086605] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2968.089570] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2968.092037] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2968.095087] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2968.097603] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2968.099739] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2968.102180] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2968.105565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2968.107797] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2968.110308] Call Trace: [ 2968.111701] [ 2968.112795] ? show_regs.cold.9+0x22/0x2f [ 2968.114111] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2968.116241] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2968.117702] ? watchdog+0x40/0x40 [ 2968.118870] ? hrtimer_interrupt+0x128/0x2c0 [ 2968.120183] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2968.121696] ? apic_timer_interrupt+0xf/0x20 [ 2968.123074] [ 2968.123628] ? native_safe_halt+0x2e/0x50 [ 2968.125161] kvm_wait+0xcd/0xe0 [ 2968.126551] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2968.128078] do_raw_spin_lock+0x177/0x190 [ 2968.129466] _raw_spin_lock+0x21/0x30 [ 2968.130855] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2968.133096] process_one_work+0x2c8/0x700 [ 2968.134408] worker_thread+0x296/0x6e0 [ 2968.136626] ? rescuer_thread+0x570/0x570 [ 2968.148794] kthread+0x1d1/0x200 [ 2968.150108] ? set_kthread_struct+0x70/0x70 [ 2968.151519] ret_from_fork+0x1f/0x30 [ 2972.048194] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 2972.061543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2972.100752] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2972.111484] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2972.117851] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2972.123937] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2972.134760] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2972.137403] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2972.141567] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 2972.147097] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 2972.150756] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 2972.156124] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 2972.158815] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 2972.162335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2972.169199] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 2972.171052] Call Trace: [ 2972.171665] [ 2972.174836] ? show_regs.cold.9+0x22/0x2f [ 2972.178929] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2972.184033] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2972.187514] ? watchdog+0x40/0x40 [ 2972.189663] ? hrtimer_interrupt+0x128/0x2c0 [ 2972.191234] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2972.198867] ? apic_timer_interrupt+0xf/0x20 [ 2972.200951] [ 2972.203562] ? native_safe_halt+0x2e/0x50 [ 2972.207288] kvm_wait+0xcd/0xe0 [ 2972.208596] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2972.212703] ? dump_stack+0xf6/0x10e [ 2972.213947] do_raw_spin_lock+0x177/0x190 [ 2972.216505] _raw_spin_lock+0x21/0x30 [ 2972.218448] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2972.221307] obd_mod_free+0x46/0x140 [ptlrpc] [ 2972.223187] mdc_commit_open+0x91/0xb0 [mdc] [ 2972.226335] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2972.229137] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2972.231872] after_reply+0xe2c/0x1780 [ptlrpc] [ 2972.235558] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2972.237916] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2972.239974] ? do_wait_intr+0xf0/0xf0 [ 2972.241319] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2972.245997] kthread+0x1d1/0x200 [ 2972.250622] ? set_kthread_struct+0x70/0x70 [ 2972.252076] ret_from_fork+0x1f/0x30 [ 2975.711049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 950s! [ 2975.714468] Showing busy workqueues and worker pools: [ 2975.716424] workqueue events: flags=0x0 [ 2975.717538] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2975.720362] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2975.722954] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2975.725028] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 2975.729055] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2975.731564] pending: kfree_rcu_monitor [ 2975.733250] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2975.736209] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 2975.740529] workqueue events_long: flags=0x0 [ 2975.741684] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2975.745075] pending: lu_objects_flush [obdclass] [ 2975.747596] workqueue events_unbound: flags=0x2 [ 2975.749922] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2975.752770] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2975.756739] workqueue events_power_efficient: flags=0x80 [ 2975.758479] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2975.760979] pending: neigh_periodic_work, check_lifetime [ 2975.763645] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2975.767396] pending: do_cache_clean, neigh_periodic_work [ 2975.769671] workqueue mm_percpu_wq: flags=0x8 [ 2975.771252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2975.773636] pending: lru_add_drain_per_cpu [ 2975.775561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2975.778502] pending: vmstat_update, lru_add_drain_per_cpu [ 2975.781064] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2975.784093] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2975.787194] workqueue writeback: flags=0x4a [ 2975.788961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2975.791787] pending: wb_workfn, wb_workfn [ 2975.793739] workqueue xprtiod: flags=0xa [ 2975.795077] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2975.797783] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2975.801048] workqueue ipv6_addrconf: flags=0x40008 [ 2975.803527] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2975.806313] pending: addrconf_verify_work [ 2975.808492] workqueue knbd0-recv: flags=0x1a [ 2975.810419] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2975.812708] in-flight: 655:recv_work [ 2975.814273] workqueue ptlrpc_pinger: flags=0x2 [ 2975.816219] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2975.818594] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 2975.820579] pool 8: cpus=0-3 flags=0x4 nice=0 hung=950s workers=4 idle: 36396 38 11188 [ 2975.823662] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2966s workers=2 idle: 658 [ 2979.363096] rcu: INFO: rcu_sched self-detected stall on CPU [ 2979.367375] rcu: 0-...!: (559439 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 2979.371666] (t=960020 jiffies g=270761 q=2907) [ 2979.374812] rcu: rcu_sched kthread starved for 954952 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 2979.378945] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2979.382998] rcu: RCU grace-period kthread stack dump: [ 2979.385102] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2979.389241] Call Trace: [ 2979.390434] __schedule+0x351/0xcb0 [ 2979.391918] schedule+0xc0/0x180 [ 2979.393193] schedule_timeout+0xb4/0x190 [ 2979.394892] ? do_raw_spin_unlock+0x75/0x190 [ 2979.396885] ? __next_timer_interrupt+0x160/0x160 [ 2979.398747] ? prepare_to_swait_event+0xf3/0x140 [ 2979.404399] rcu_gp_kthread+0x8dd/0x15b0 [ 2979.405830] ? force_qs_rnp+0x2c0/0x2c0 [ 2979.407949] kthread+0x1d1/0x200 [ 2979.411473] ? set_kthread_struct+0x70/0x70 [ 2979.413982] ret_from_fork+0x1f/0x30 [ 2979.415638] rcu: Stack dump where RCU GP kthread last ran: [ 2979.418281] Sending NMI from CPU 0 to CPUs 2: [ 2979.431165] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 4.963 msecs [ 2979.431202] NMI backtrace for cpu 2 [ 2979.431210] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2979.431213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2979.431215] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2979.431216] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2979.431219] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 [ 2979.431223] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 2979.431224] RDX: 0000000000005b99 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 2979.431226] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 2979.431227] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 2979.431229] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2979.431231] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 2979.431232] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2979.431233] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 2979.431235] Call Trace: [ 2979.431236] [ 2979.431237] ? show_regs.cold.9+0x22/0x2f [ 2979.431238] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2979.431239] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2979.431240] ? nmi_handle+0xbd/0x210 [ 2979.431241] ? default_do_nmi+0x4b/0x1b0 [ 2979.431242] ? do_nmi+0x280/0x380 [ 2979.431244] ? end_repeat_nmi+0x16/0x1f [ 2979.431245] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2979.431246] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2979.431247] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2979.431248] [ 2979.431250] do_raw_spin_lock+0x177/0x190 [ 2979.431251] _raw_spin_lock+0x21/0x30 [ 2979.431252] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2979.431254] ? do_raw_spin_unlock+0x75/0x190 [ 2979.431254] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2979.431256] ? do_wait_intr+0xf0/0xf0 [ 2979.431257] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2979.431258] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2979.431260] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2979.431261] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2979.431263] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2979.431264] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2979.431265] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2979.431266] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2979.431524] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2979.431525] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2979.431527] lmv_intent_open+0x338/0x1150 [lmv] [ 2979.431528] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2979.431529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2979.431530] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2979.431531] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2979.431532] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2979.431533] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2979.431534] ? do_raw_spin_unlock+0x75/0x190 [ 2979.431535] ? _raw_spin_unlock+0x12/0x30 [ 2979.431536] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2979.431537] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2979.431539] ll_atomic_open+0x36a/0xd70 [lustre] [ 2979.431540] lookup_open+0x356/0xb40 [ 2979.431541] do_last+0x313/0xfc0 [ 2979.431542] ? nd_jump_root+0xe5/0x160 [ 2979.431543] ? path_init+0x437/0x520 [ 2979.431544] path_openat+0xf7/0x500 [ 2979.431545] ? do_raw_spin_unlock+0x75/0x190 [ 2979.431546] do_filp_open+0x99/0x140 [ 2979.431547] ? getname_flags+0x6e/0x330 [ 2979.431548] ? __check_object_size+0xff/0x256 [ 2979.431550] ? do_raw_spin_unlock+0x75/0x190 [ 2979.431551] ? _raw_spin_unlock+0x12/0x30 [ 2979.431552] do_sys_openat2+0x2b4/0x410 [ 2979.431553] do_sys_open+0x73/0xa0 [ 2979.431555] __x64_sys_openat+0x24/0x30 [ 2979.431555] do_syscall_64+0xc1/0x440 [ 2979.431557] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2979.431558] RIP: 0033:0x7fdc0afbf332 [ 2979.431559] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2979.431561] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2979.431564] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 2979.431566] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 2979.431567] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 2979.431569] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 2979.431570] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2979.431577] NMI backtrace for cpu 0 [ 2979.637551] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2979.643610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2979.648347] Call Trace: [ 2979.649372] [ 2979.650274] ? dump_stack+0xbb/0x10e [ 2979.652273] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2979.654476] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2979.656740] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2979.659063] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2979.661147] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2979.663415] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2979.665364] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2979.667577] ? update_process_times+0x6d/0xc0 [ 2979.669696] ? tick_sched_handle.isra.18+0x27/0x90 [ 2979.671654] ? tick_sched_timer+0x53/0xb0 [ 2979.673352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2979.675231] ? tick_sched_handle.isra.18+0x90/0x90 [ 2979.677341] ? hrtimer_interrupt+0x128/0x2c0 [ 2979.679052] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2979.681102] ? apic_timer_interrupt+0xf/0x20 [ 2979.682955] [ 2979.683923] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2979.686167] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2979.688481] ? dump_stack+0xf6/0x10e [ 2979.690025] ? do_raw_spin_lock+0x177/0x190 [ 2979.691692] ? _raw_spin_lock+0x21/0x30 [ 2979.693233] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2979.695835] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2979.698231] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2979.700495] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2979.703103] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2979.706483] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2979.709012] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2979.712088] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2979.714977] ? do_wait_intr+0xf0/0xf0 [ 2979.716830] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2979.720854] ? kthread+0x1d1/0x200 [ 2979.722902] ? set_kthread_struct+0x70/0x70 [ 2979.724893] ? ret_from_fork+0x1f/0x30 [ 2980.057701] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 2980.062872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2980.091239] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2980.097722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2980.101631] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2980.104237] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2980.115403] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2980.123023] RAX: 000000000000578d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2980.124726] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 2980.132914] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2980.135894] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 2980.139493] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 2980.143634] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 2980.148046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2980.150622] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 2980.154744] Call Trace: [ 2980.156105] [ 2980.157133] ? show_regs.cold.9+0x22/0x2f [ 2980.159208] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2980.161839] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2980.168383] ? watchdog+0x40/0x40 [ 2980.170361] ? hrtimer_interrupt+0x128/0x2c0 [ 2980.176142] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2980.181699] ? apic_timer_interrupt+0xf/0x20 [ 2980.184368] [ 2980.187545] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2980.192452] do_raw_spin_lock+0x177/0x190 [ 2980.196300] _raw_spin_lock+0x21/0x30 [ 2980.197269] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2980.201463] ? do_raw_spin_unlock+0x75/0x190 [ 2980.205137] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2980.208412] ? do_wait_intr+0xf0/0xf0 [ 2980.210635] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2980.213595] mdc_close+0x53b/0xd70 [mdc] [ 2980.215477] lmv_close+0x18d/0x490 [lmv] [ 2980.217882] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2980.220797] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2980.223609] ll_md_close+0x274/0x800 [lustre] [ 2980.228754] ll_file_release+0x128/0xc50 [lustre] [ 2980.234603] __fput+0x10d/0x440 [ 2980.237854] ____fput+0x12/0x20 [ 2980.239198] task_work_run+0xc6/0x110 [ 2980.241622] exit_to_usermode_loop+0x1dd/0x1f0 [ 2980.243180] do_syscall_64+0x3d6/0x440 [ 2980.245338] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2980.248026] RIP: 0033:0x7fdc0afbfcd8 [ 2980.249924] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2980.258792] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2980.262581] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 2980.266235] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 2980.268598] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 2980.270929] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 2980.274452] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2996.051025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 2996.055233] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2996.080230] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2996.089315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2996.096999] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2996.107258] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2996.114639] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2996.131780] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2996.135270] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 2996.140971] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 2996.147527] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2996.152327] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 2996.156628] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 2996.161950] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 2996.166695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2996.169267] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 2996.174581] Call Trace: [ 2996.178141] [ 2996.178931] ? show_regs.cold.9+0x22/0x2f [ 2996.180738] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2996.185375] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2996.186857] ? watchdog+0x40/0x40 [ 2996.190936] ? hrtimer_interrupt+0x128/0x2c0 [ 2996.192453] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2996.195831] ? apic_timer_interrupt+0xf/0x20 [ 2996.197715] [ 2996.199243] ? native_safe_halt+0x2e/0x50 [ 2996.201232] kvm_wait+0xcd/0xe0 [ 2996.202654] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2996.207851] do_raw_spin_lock+0x177/0x190 [ 2996.212524] _raw_spin_lock+0x21/0x30 [ 2996.215598] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2996.218232] process_one_work+0x2c8/0x700 [ 2996.220220] worker_thread+0x296/0x6e0 [ 2996.222223] ? rescuer_thread+0x570/0x570 [ 2996.225932] kthread+0x1d1/0x200 [ 2996.229327] ? set_kthread_struct+0x70/0x70 [ 2996.232589] ret_from_fork+0x1f/0x30 [ 3004.050240] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3004.052500] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3004.056546] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3004.056705] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3004.056713] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3004.056715] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3004.056728] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3004.056733] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3004.056737] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3004.056740] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3004.056742] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3004.056744] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3004.056746] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3004.056750] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3004.056753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3004.056755] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3004.056763] Call Trace: [ 3004.056769] [ 3004.056775] ? show_regs.cold.9+0x22/0x2f [ 3004.056784] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3004.056791] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3004.056795] ? watchdog+0x40/0x40 [ 3004.056799] ? hrtimer_interrupt+0x128/0x2c0 [ 3004.056803] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3004.056810] ? apic_timer_interrupt+0xf/0x20 [ 3004.056814] [ 3004.056818] ? native_safe_halt+0x2e/0x50 [ 3004.056833] kvm_wait+0xcd/0xe0 [ 3004.056943] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3004.056950] ? dump_stack+0xf6/0x10e [ 3004.056957] do_raw_spin_lock+0x177/0x190 [ 3004.056961] _raw_spin_lock+0x21/0x30 [ 3004.056967] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3004.063452] Modules linked in: lustre(O) [ 3004.103539] obd_mod_free+0x46/0x140 [ptlrpc] [ 3004.112291] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3004.112364] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3004.112371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3004.112374] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3004.112388] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3004.112392] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3004.112397] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3004.112399] RDX: 0000000000001d9a RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3004.112401] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3004.112403] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3004.112405] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3004.112409] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3004.112411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3004.112414] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3004.112423] Call Trace: [ 3004.112428] [ 3004.112434] ? show_regs.cold.9+0x22/0x2f [ 3004.112443] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3004.112450] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3004.112455] ? watchdog+0x40/0x40 [ 3004.112459] ? hrtimer_interrupt+0x128/0x2c0 [ 3004.112464] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3004.112472] ? apic_timer_interrupt+0xf/0x20 [ 3004.112477] [ 3004.112481] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3004.112487] do_raw_spin_lock+0x177/0x190 [ 3004.112491] _raw_spin_lock+0x21/0x30 [ 3004.112497] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3004.112778] ? do_raw_spin_unlock+0x75/0x190 [ 3004.112784] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3004.122322] mdc_commit_open+0x91/0xb0 [mdc] [ 3004.122374] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3004.122661] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3004.122885] after_reply+0xe2c/0x1780 [ptlrpc] [ 3004.123104] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3004.123335] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3004.123560] ? do_wait_intr+0xf0/0xf0 [ 3004.123571] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3004.123792] kthread+0x1d1/0x200 [ 3004.123798] ? set_kthread_struct+0x70/0x70 [ 3004.123803] ret_from_fork+0x1f/0x30 [ 3004.500654] ? do_wait_intr+0xf0/0xf0 [ 3004.504629] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3004.512381] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3004.514738] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3004.523469] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3004.528725] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3004.536218] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3004.539760] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3004.541904] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3004.545455] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3004.550159] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3004.557173] lmv_intent_open+0x338/0x1150 [lmv] [ 3004.560278] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3004.567476] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3004.570631] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3004.575793] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3004.578429] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3004.585559] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3004.589488] ? do_raw_spin_unlock+0x75/0x190 [ 3004.590728] ? _raw_spin_unlock+0x12/0x30 [ 3004.593766] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3004.597629] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3004.598847] ll_atomic_open+0x36a/0xd70 [lustre] [ 3004.601961] lookup_open+0x356/0xb40 [ 3004.606492] do_last+0x313/0xfc0 [ 3004.609520] ? nd_jump_root+0xe5/0x160 [ 3004.610629] ? path_init+0x437/0x520 [ 3004.611596] path_openat+0xf7/0x500 [ 3004.622924] ? do_raw_spin_unlock+0x75/0x190 [ 3004.624632] do_filp_open+0x99/0x140 [ 3004.626458] ? getname_flags+0x6e/0x330 [ 3004.627931] ? __check_object_size+0xff/0x256 [ 3004.629942] ? do_raw_spin_unlock+0x75/0x190 [ 3004.631739] ? _raw_spin_unlock+0x12/0x30 [ 3004.633359] do_sys_openat2+0x2b4/0x410 [ 3004.635497] do_sys_open+0x73/0xa0 [ 3004.644089] __x64_sys_openat+0x24/0x30 [ 3004.649466] do_syscall_64+0xc1/0x440 [ 3004.650648] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3004.655278] RIP: 0033:0x7fdc0afbf332 [ 3004.656588] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3004.663558] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3004.666254] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3004.668719] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3004.671568] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3004.674185] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3004.676692] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3005.921014] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 980s! [ 3005.926750] Showing busy workqueues and worker pools: [ 3005.930087] workqueue events: flags=0x0 [ 3005.935638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3005.940092] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3005.945340] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3005.952315] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3005.963262] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3005.969081] pending: kfree_rcu_monitor [ 3005.973941] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3005.981202] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3005.997233] workqueue events_long: flags=0x0 [ 3005.998803] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3006.006243] pending: lu_objects_flush [obdclass] [ 3006.010341] workqueue events_unbound: flags=0x2 [ 3006.012418] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3006.014716] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3006.018296] workqueue events_power_efficient: flags=0x80 [ 3006.020671] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3006.024055] pending: neigh_periodic_work, check_lifetime [ 3006.027901] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3006.037471] pending: do_cache_clean, neigh_periodic_work [ 3006.040254] workqueue mm_percpu_wq: flags=0x8 [ 3006.042108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3006.045459] pending: lru_add_drain_per_cpu [ 3006.047677] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3006.050520] pending: vmstat_update, lru_add_drain_per_cpu [ 3006.053154] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3006.056094] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3006.058445] workqueue writeback: flags=0x4a [ 3006.059738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3006.061412] pending: wb_workfn, wb_workfn [ 3006.063062] workqueue xprtiod: flags=0xa [ 3006.064936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3006.067880] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3006.078043] workqueue ipv6_addrconf: flags=0x40008 [ 3006.080495] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3006.083930] pending: addrconf_verify_work [ 3006.086237] workqueue knbd0-recv: flags=0x1a [ 3006.088086] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3006.091014] in-flight: 655:recv_work [ 3006.092873] workqueue ptlrpc_pinger: flags=0x2 [ 3006.094669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3006.097355] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3006.100042] pool 8: cpus=0-3 flags=0x4 nice=0 hung=981s workers=4 idle: 36396 38 11188 [ 3006.107126] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2996s workers=2 idle: 658 [ 3008.054213] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3008.061477] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3008.090705] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3008.095068] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3008.098927] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3008.100925] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3008.109145] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3008.112329] RAX: 0000000000005f18 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3008.115106] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3008.118197] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3008.120521] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3008.123170] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3008.126730] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3008.130342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3008.133319] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3008.136586] Call Trace: [ 3008.137699] [ 3008.138975] ? show_regs.cold.9+0x22/0x2f [ 3008.142124] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3008.144877] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3008.147356] ? watchdog+0x40/0x40 [ 3008.149145] ? hrtimer_interrupt+0x128/0x2c0 [ 3008.151140] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3008.152881] ? apic_timer_interrupt+0xf/0x20 [ 3008.154244] [ 3008.154930] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3008.156913] do_raw_spin_lock+0x177/0x190 [ 3008.158191] _raw_spin_lock+0x21/0x30 [ 3008.159362] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3008.161366] ? do_raw_spin_unlock+0x75/0x190 [ 3008.162752] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3008.164728] ? do_wait_intr+0xf0/0xf0 [ 3008.166026] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3008.168092] mdc_close+0x53b/0xd70 [mdc] [ 3008.169669] lmv_close+0x18d/0x490 [lmv] [ 3008.171215] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3008.173408] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3008.175194] ll_md_close+0x274/0x800 [lustre] [ 3008.177440] ll_file_release+0x128/0xc50 [lustre] [ 3008.179447] __fput+0x10d/0x440 [ 3008.180905] ____fput+0x12/0x20 [ 3008.182201] task_work_run+0xc6/0x110 [ 3008.183659] exit_to_usermode_loop+0x1dd/0x1f0 [ 3008.185479] do_syscall_64+0x3d6/0x440 [ 3008.187119] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3008.189149] RIP: 0033:0x7fdc0afbfcd8 [ 3008.190831] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3008.197739] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3008.199998] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3008.202601] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3008.205419] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3008.208429] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3008.211106] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3024.051521] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3024.059994] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3024.083632] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3024.093211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3024.100985] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3024.106740] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3024.113132] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3024.131804] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3024.137143] RAX: 0000000000007f88 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3024.146756] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3024.155004] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3024.162227] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3024.166577] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3024.173100] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3024.180691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3024.185515] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3024.188376] Call Trace: [ 3024.189127] [ 3024.190049] ? show_regs.cold.9+0x22/0x2f [ 3024.191811] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3024.194100] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3024.195880] ? watchdog+0x40/0x40 [ 3024.197148] ? hrtimer_interrupt+0x128/0x2c0 [ 3024.198787] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3024.201062] ? apic_timer_interrupt+0xf/0x20 [ 3024.205252] [ 3024.206078] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3024.210024] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3024.212029] do_raw_spin_lock+0x177/0x190 [ 3024.213684] _raw_spin_lock+0x21/0x30 [ 3024.214730] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3024.216648] process_one_work+0x2c8/0x700 [ 3024.219747] worker_thread+0x296/0x6e0 [ 3024.220852] ? rescuer_thread+0x570/0x570 [ 3024.222095] kthread+0x1d1/0x200 [ 3024.223396] ? set_kthread_struct+0x70/0x70 [ 3024.224799] ret_from_fork+0x1f/0x30 [ 3032.048079] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3032.051716] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3032.054585] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3032.073521] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3032.073535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3032.073539] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3032.073553] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3032.073575] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3032.073589] RAX: 000000000000410e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3032.073592] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3032.073594] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3032.073596] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3032.073598] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3032.073601] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3032.073605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3032.073607] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3032.073617] Call Trace: [ 3032.073622] [ 3032.073627] ? show_regs.cold.9+0x22/0x2f [ 3032.073640] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3032.073647] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3032.073655] ? watchdog+0x40/0x40 [ 3032.073659] ? hrtimer_interrupt+0x128/0x2c0 [ 3032.073663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3032.073677] ? apic_timer_interrupt+0xf/0x20 [ 3032.073681] [ 3032.073683] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3032.073687] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3032.073695] ? dump_stack+0xf6/0x10e [ 3032.073701] do_raw_spin_lock+0x177/0x190 [ 3032.073709] _raw_spin_lock+0x21/0x30 [ 3032.073717] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3032.074223] obd_mod_free+0x46/0x140 [ptlrpc] [ 3032.074420] mdc_commit_open+0x91/0xb0 [mdc] [ 3032.074595] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3032.074805] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3032.075161] after_reply+0xe2c/0x1780 [ptlrpc] [ 3032.075376] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3032.075602] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3032.075848] ? do_wait_intr+0xf0/0xf0 [ 3032.075863] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3032.083993] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3032.084075] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3032.084083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3032.084085] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3032.084101] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3032.090216] kthread+0x1d1/0x200 [ 3032.090240] ? set_kthread_struct+0x70/0x70 [ 3032.090246] ret_from_fork+0x1f/0x30 [ 3032.096303] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3032.198747] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3032.200946] RDX: 00000000000033cd RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3032.202842] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3032.204578] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3032.206304] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3032.208247] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3032.210654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3032.212235] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3032.214218] Call Trace: [ 3032.214937] [ 3032.215470] ? show_regs.cold.9+0x22/0x2f [ 3032.216593] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3032.217958] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3032.219150] ? watchdog+0x40/0x40 [ 3032.219913] ? hrtimer_interrupt+0x128/0x2c0 [ 3032.221008] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3032.222214] ? apic_timer_interrupt+0xf/0x20 [ 3032.223280] [ 3032.224482] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3032.225895] do_raw_spin_lock+0x177/0x190 [ 3032.226925] _raw_spin_lock+0x21/0x30 [ 3032.227902] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3032.229481] ? do_raw_spin_unlock+0x75/0x190 [ 3032.230678] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3032.232233] ? do_wait_intr+0xf0/0xf0 [ 3032.233318] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3032.234852] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3032.236263] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3032.238207] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3032.239748] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3032.241306] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3032.242577] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3032.243609] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3032.245205] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3032.246963] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3032.248506] lmv_intent_open+0x338/0x1150 [lmv] [ 3032.249703] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3032.251264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3032.252642] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3032.253804] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3032.255021] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3032.256566] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3032.257826] ? do_raw_spin_unlock+0x75/0x190 [ 3032.258949] ? _raw_spin_unlock+0x12/0x30 [ 3032.259957] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3032.261344] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3032.262688] ll_atomic_open+0x36a/0xd70 [lustre] [ 3032.264121] lookup_open+0x356/0xb40 [ 3032.265091] do_last+0x313/0xfc0 [ 3032.265935] ? nd_jump_root+0xe5/0x160 [ 3032.266898] ? path_init+0x437/0x520 [ 3032.267821] path_openat+0xf7/0x500 [ 3032.268676] ? do_raw_spin_unlock+0x75/0x190 [ 3032.269766] do_filp_open+0x99/0x140 [ 3032.270707] ? getname_flags+0x6e/0x330 [ 3032.271667] ? __check_object_size+0xff/0x256 [ 3032.272802] ? do_raw_spin_unlock+0x75/0x190 [ 3032.273930] ? _raw_spin_unlock+0x12/0x30 [ 3032.274939] do_sys_openat2+0x2b4/0x410 [ 3032.276096] do_sys_open+0x73/0xa0 [ 3032.277048] __x64_sys_openat+0x24/0x30 [ 3032.278115] do_syscall_64+0xc1/0x440 [ 3032.279166] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3032.280604] RIP: 0033:0x7fdc0afbf332 [ 3032.281661] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3032.286655] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3032.288734] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3032.290710] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3032.292656] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3032.294774] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3032.296915] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3036.054024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3036.056299] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3036.087673] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3036.092471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3036.096221] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3036.098187] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3036.105742] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3036.108215] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3036.111060] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3036.114387] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3036.117440] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3036.120112] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3036.123265] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3036.127051] BUG: workqueue lockup - pool [ 3036.127380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3036.127382] cpus=0-3 flags=0x4 nice=0 [ 3036.129003] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3036.129018] Call Trace: [ 3036.129026] [ 3036.129032] ? show_regs.cold.9+0x22/0x2f [ 3036.134132] stuck for 1011s! [ 3036.134150] Showing busy workqueues and worker pools: [ 3036.134153] workqueue events: flags=0x0 [ 3036.134182] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3036.134448] pending: cpu_latency_work [ptlrpc] [ 3036.136816] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3036.136831] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3036.136836] ? watchdog+0x40/0x40 [ 3036.136840] ? hrtimer_interrupt+0x128/0x2c0 [ 3036.136844] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3036.136854] ? apic_timer_interrupt+0xf/0x20 [ 3036.136872] [ 3036.136877] ? native_safe_halt+0x2e/0x50 [ 3036.136882] kvm_wait+0xcd/0xe0 [ 3036.136892] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3036.136901] do_raw_spin_lock+0x177/0x190 [ 3036.136906] _raw_spin_lock+0x21/0x30 [ 3036.136911] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3036.141269] , push_to_pool [ 3036.141853] ? do_raw_spin_unlock+0x75/0x190 [ 3036.145358] [ 3036.145391] pwq 4: [ 3036.147181] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3036.147533] ? do_wait_intr+0xf0/0xf0 [ 3036.147545] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3036.147764] mdc_close+0x53b/0xd70 [mdc] [ 3036.147801] lmv_close+0x18d/0x490 [lmv] [ 3036.147835] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3036.147971] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3036.148062] ll_md_close+0x274/0x800 [lustre] [ 3036.151214] cpus=2 [ 3036.153270] ll_file_release+0x128/0xc50 [lustre] [ 3036.153391] __fput+0x10d/0x440 [ 3036.153400] ____fput+0x12/0x20 [ 3036.153404] task_work_run+0xc6/0x110 [ 3036.153413] exit_to_usermode_loop+0x1dd/0x1f0 [ 3036.153420] do_syscall_64+0x3d6/0x440 [ 3036.153424] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3036.153433] RIP: 0033:0x7fdc0afbfcd8 [ 3036.153440] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3036.153445] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 [ 3036.156741] node=0 [ 3036.161236] ORIG_RAX: 0000000000000003 [ 3036.161246] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3036.161248] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3036.161251] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3036.161253] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3036.161255] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3036.293860] flags=0x0 nice=0 active=3/256 refcnt=4 [ 3036.295896] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3036.298643] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3036.302974] pending: kfree_rcu_monitor [ 3036.304189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3036.306413] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3036.311931] workqueue events_long: flags=0x0 [ 3036.314613] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3036.317088] pending: lu_objects_flush [obdclass] [ 3036.319910] workqueue events_unbound: flags=0x2 [ 3036.321381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3036.323799] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3036.329211] workqueue events_power_efficient: flags=0x80 [ 3036.332117] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3036.335167] pending: neigh_periodic_work, check_lifetime [ 3036.337633] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3036.340831] pending: do_cache_clean, neigh_periodic_work [ 3036.343350] workqueue mm_percpu_wq: flags=0x8 [ 3036.344555] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3036.349211] pending: lru_add_drain_per_cpu [ 3036.351356] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3036.353718] pending: vmstat_update, lru_add_drain_per_cpu [ 3036.355270] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3036.358607] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3036.362641] workqueue writeback: flags=0x4a [ 3036.364394] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3036.366914] pending: wb_workfn, wb_workfn [ 3036.371263] workqueue xprtiod: flags=0xa [ 3036.373344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3036.375870] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3036.379796] workqueue ipv6_addrconf: flags=0x40008 [ 3036.381593] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3036.388016] pending: addrconf_verify_work [ 3036.389218] workqueue knbd0-recv: flags=0x1a [ 3036.390934] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3036.394817] in-flight: 655:recv_work [ 3036.396167] workqueue ptlrpc_pinger: flags=0x2 [ 3036.398410] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3036.400240] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3036.407213] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1011s workers=4 idle: 36396 38 11188 [ 3036.411774] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3026s workers=2 idle: 658 [ 3052.053428] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3052.058196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3052.107350] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3052.117766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3052.125725] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3052.130764] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3052.135776] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3052.148951] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3052.154977] RAX: 0000000000006e3b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3052.162873] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3052.170739] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3052.178795] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3052.190756] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3052.195974] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3052.204058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3052.208997] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3052.221803] Call Trace: [ 3052.224195] [ 3052.224995] ? show_regs.cold.9+0x22/0x2f [ 3052.228368] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3052.237462] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3052.241081] ? watchdog+0x40/0x40 [ 3052.243985] ? hrtimer_interrupt+0x128/0x2c0 [ 3052.246541] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3052.251598] ? apic_timer_interrupt+0xf/0x20 [ 3052.256079] [ 3052.256896] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3052.260791] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3052.265829] do_raw_spin_lock+0x177/0x190 [ 3052.270755] _raw_spin_lock+0x21/0x30 [ 3052.273898] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3052.280474] process_one_work+0x2c8/0x700 [ 3052.285999] worker_thread+0x296/0x6e0 [ 3052.292367] ? rescuer_thread+0x570/0x570 [ 3052.297666] kthread+0x1d1/0x200 [ 3052.302123] ? set_kthread_struct+0x70/0x70 [ 3052.303638] ret_from_fork+0x1f/0x30 [ 3060.057871] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3060.066739] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3060.092729] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3060.098166] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3060.102158] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3060.105105] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3060.119140] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3060.123056] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3060.126065] RDX: 0000000000007d1d RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3060.135095] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3060.146040] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3060.151166] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3060.154627] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3060.160810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3060.165871] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3060.169768] Call Trace: [ 3060.170667] [ 3060.172833] ? show_regs.cold.9+0x22/0x2f [ 3060.174674] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3060.177699] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3060.181324] ? watchdog+0x40/0x40 [ 3060.182294] ? hrtimer_interrupt+0x128/0x2c0 [ 3060.185311] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3060.186858] ? apic_timer_interrupt+0xf/0x20 [ 3060.190368] [ 3060.191380] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3060.193288] do_raw_spin_lock+0x177/0x190 [ 3060.194543] _raw_spin_lock+0x21/0x30 [ 3060.195746] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3060.199085] ? do_raw_spin_unlock+0x75/0x190 [ 3060.200793] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3060.203467] ? do_wait_intr+0xf0/0xf0 [ 3060.205493] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3060.208043] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3060.210822] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3060.215581] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3060.218224] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3060.222920] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3060.225638] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3060.228339] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3060.231096] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3060.234036] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3060.239723] lmv_intent_open+0x338/0x1150 [lmv] [ 3060.247805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3060.255606] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3060.257696] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3060.259237] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3060.260822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3060.264137] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3060.266501] ? do_raw_spin_unlock+0x75/0x190 [ 3060.268421] ? _raw_spin_unlock+0x12/0x30 [ 3060.270089] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3060.272246] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3060.274540] ll_atomic_open+0x36a/0xd70 [lustre] [ 3060.276921] lookup_open+0x356/0xb40 [ 3060.278599] do_last+0x313/0xfc0 [ 3060.280503] ? nd_jump_root+0xe5/0x160 [ 3060.283756] ? path_init+0x437/0x520 [ 3060.286284] path_openat+0xf7/0x500 [ 3060.287534] ? do_raw_spin_unlock+0x75/0x190 [ 3060.288956] do_filp_open+0x99/0x140 [ 3060.290112] ? getname_flags+0x6e/0x330 [ 3060.291787] ? __check_object_size+0xff/0x256 [ 3060.294128] ? do_raw_spin_unlock+0x75/0x190 [ 3060.296126] ? _raw_spin_unlock+0x12/0x30 [ 3060.297527] do_sys_openat2+0x2b4/0x410 [ 3060.298763] do_sys_open+0x73/0xa0 [ 3060.302062] __x64_sys_openat+0x24/0x30 [ 3060.304165] do_syscall_64+0xc1/0x440 [ 3060.307491] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3060.311324] RIP: 0033:0x7fdc0afbf332 [ 3060.314519] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3060.323656] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3060.327528] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3060.330827] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3060.335132] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3060.341622] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3060.350538] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3064.052958] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3064.055399] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3064.058376] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3064.082711] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3064.082722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3064.082725] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3064.086722] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3064.086798] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3064.086805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3064.086807] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3064.086822] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3064.086826] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3064.086831] RAX: 0000000000002116 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3064.086833] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3064.086835] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3064.086837] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3064.086839] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3064.086841] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3064.086843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3064.086845] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3064.086855] Call Trace: [ 3064.086860] [ 3064.086866] ? show_regs.cold.9+0x22/0x2f [ 3064.086875] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3064.086881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3064.086885] ? watchdog+0x40/0x40 [ 3064.086890] ? hrtimer_interrupt+0x128/0x2c0 [ 3064.086894] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3064.086903] ? apic_timer_interrupt+0xf/0x20 [ 3064.086907] [ 3064.086910] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3064.086916] do_raw_spin_lock+0x177/0x190 [ 3064.086921] _raw_spin_lock+0x21/0x30 [ 3064.086926] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3064.087284] ? do_raw_spin_unlock+0x75/0x190 [ 3064.087290] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3064.087599] ? do_wait_intr+0xf0/0xf0 [ 3064.087606] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3064.087817] mdc_close+0x53b/0xd70 [mdc] [ 3064.087855] lmv_close+0x18d/0x490 [lmv] [ 3064.087883] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3064.088004] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3064.088226] ll_md_close+0x274/0x800 [lustre] [ 3064.088313] ll_file_release+0x128/0xc50 [lustre] [ 3064.088394] __fput+0x10d/0x440 [ 3064.088403] ____fput+0x12/0x20 [ 3064.088406] task_work_run+0xc6/0x110 [ 3064.088413] exit_to_usermode_loop+0x1dd/0x1f0 [ 3064.088420] do_syscall_64+0x3d6/0x440 [ 3064.088424] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3064.088430] RIP: 0033:0x7fdc0afbfcd8 [ 3064.088437] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3064.088498] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3064.088504] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3064.088506] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3064.088508] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3064.088510] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3064.088512] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3064.287879] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3064.298851] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3064.305439] RAX: 0000000000007c59 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3064.311873] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3064.317077] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3064.325502] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3064.333214] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3064.338094] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3064.341376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3064.343619] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3064.346626] Call Trace: [ 3064.347608] [ 3064.348422] ? show_regs.cold.9+0x22/0x2f [ 3064.350445] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3064.352453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3064.354300] ? watchdog+0x40/0x40 [ 3064.355617] ? hrtimer_interrupt+0x128/0x2c0 [ 3064.357360] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3064.359380] ? apic_timer_interrupt+0xf/0x20 [ 3064.361229] [ 3064.362109] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3064.364436] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3064.366668] ? dump_stack+0xf6/0x10e [ 3064.368288] do_raw_spin_lock+0x177/0x190 [ 3064.370057] _raw_spin_lock+0x21/0x30 [ 3064.371633] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3064.374665] obd_mod_free+0x46/0x140 [ptlrpc] [ 3064.377404] mdc_commit_open+0x91/0xb0 [mdc] [ 3064.380965] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3064.387386] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3064.395312] after_reply+0xe2c/0x1780 [ptlrpc] [ 3064.397489] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3064.403294] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3064.406057] ? do_wait_intr+0xf0/0xf0 [ 3064.409546] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3064.418319] kthread+0x1d1/0x200 [ 3064.422100] ? set_kthread_struct+0x70/0x70 [ 3064.426240] ret_from_fork+0x1f/0x30 [ 3066.848206] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1041s! [ 3066.858239] Showing busy workqueues and worker pools: [ 3066.860847] workqueue events: flags=0x0 [ 3066.863228] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3066.881572] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3066.884519] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3066.887517] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3066.896023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3066.902180] pending: kfree_rcu_monitor [ 3066.904268] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3066.909815] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3066.914335] workqueue events_long: flags=0x0 [ 3066.919277] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3066.923269] pending: lu_objects_flush [obdclass] [ 3066.925693] workqueue events_unbound: flags=0x2 [ 3066.927713] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3066.929992] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3066.934717] workqueue events_power_efficient: flags=0x80 [ 3066.936606] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3066.942090] pending: neigh_periodic_work, check_lifetime [ 3066.944450] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3066.947475] pending: do_cache_clean, neigh_periodic_work [ 3066.951628] workqueue mm_percpu_wq: flags=0x8 [ 3066.959019] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3066.966551] pending: lru_add_drain_per_cpu [ 3066.968977] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3066.970852] pending: vmstat_update, lru_add_drain_per_cpu [ 3066.976339] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3066.988027] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3066.997693] workqueue writeback: flags=0x4a [ 3066.999265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3067.006330] pending: wb_workfn, wb_workfn [ 3067.008423] workqueue xprtiod: flags=0xa [ 3067.010278] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3067.014481] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3067.021448] workqueue ipv6_addrconf: flags=0x40008 [ 3067.023659] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3067.026646] pending: addrconf_verify_work [ 3067.028477] workqueue knbd0-recv: flags=0x1a [ 3067.032692] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3067.038482] in-flight: 655:recv_work [ 3067.041070] workqueue ptlrpc_pinger: flags=0x2 [ 3067.042815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3067.045253] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3067.048207] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1042s workers=4 idle: 36396 38 11188 [ 3067.052514] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3057s workers=2 idle: 658 [ 3080.056157] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3080.068827] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3080.112823] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3080.124724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3080.130734] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3080.136996] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3080.143983] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3080.157853] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3080.163751] RAX: 0000000000004a69 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3080.169153] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3080.176290] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3080.186013] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3080.192977] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3080.200240] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3080.213685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3080.223999] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3080.229501] Call Trace: [ 3080.230845] [ 3080.231642] ? show_regs.cold.9+0x22/0x2f [ 3080.242348] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3080.260364] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3080.268606] ? watchdog+0x40/0x40 [ 3080.269855] ? hrtimer_interrupt+0x128/0x2c0 [ 3080.282308] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3080.289235] ? apic_timer_interrupt+0xf/0x20 [ 3080.296361] [ 3080.303343] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3080.308810] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3080.321762] do_raw_spin_lock+0x177/0x190 [ 3080.323257] _raw_spin_lock+0x21/0x30 [ 3080.328676] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3080.334140] process_one_work+0x2c8/0x700 [ 3080.338343] worker_thread+0x296/0x6e0 [ 3080.341605] ? rescuer_thread+0x570/0x570 [ 3080.343689] kthread+0x1d1/0x200 [ 3080.344883] ? set_kthread_struct+0x70/0x70 [ 3080.347761] ret_from_fork+0x1f/0x30 [ 3088.052035] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3088.055114] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3088.112043] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3088.123207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3088.130242] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3088.134290] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3088.144752] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3088.148700] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3088.151287] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3088.154876] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3088.162840] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3088.168743] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3088.173540] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3088.178480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3088.182313] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3088.190847] Call Trace: [ 3088.193307] [ 3088.194987] ? show_regs.cold.9+0x22/0x2f [ 3088.197923] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3088.201918] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3088.204675] ? watchdog+0x40/0x40 [ 3088.208449] ? hrtimer_interrupt+0x128/0x2c0 [ 3088.215575] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3088.224525] ? apic_timer_interrupt+0xf/0x20 [ 3088.229304] [ 3088.231467] ? native_safe_halt+0x2e/0x50 [ 3088.237677] kvm_wait+0xcd/0xe0 [ 3088.240253] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3088.249682] do_raw_spin_lock+0x177/0x190 [ 3088.253550] _raw_spin_lock+0x21/0x30 [ 3088.256467] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3088.262315] ? do_raw_spin_unlock+0x75/0x190 [ 3088.265282] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3088.269306] ? do_wait_intr+0xf0/0xf0 [ 3088.272867] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3088.275678] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3088.281564] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3088.285851] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3088.291901] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3088.294940] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3088.297585] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3088.300760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3088.312823] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3088.333473] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3088.342786] lmv_intent_open+0x338/0x1150 [lmv] [ 3088.347550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3088.357658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3088.363839] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3088.369631] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3088.373886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3088.379334] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3088.384701] ? do_raw_spin_unlock+0x75/0x190 [ 3088.388558] ? _raw_spin_unlock+0x12/0x30 [ 3088.396453] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3088.400688] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3088.406261] ll_atomic_open+0x36a/0xd70 [lustre] [ 3088.408196] lookup_open+0x356/0xb40 [ 3088.423864] do_last+0x313/0xfc0 [ 3088.425852] ? nd_jump_root+0xe5/0x160 [ 3088.435029] ? path_init+0x437/0x520 [ 3088.436218] path_openat+0xf7/0x500 [ 3088.440720] ? do_raw_spin_unlock+0x75/0x190 [ 3088.453830] do_filp_open+0x99/0x140 [ 3088.458455] ? getname_flags+0x6e/0x330 [ 3088.462511] ? __check_object_size+0xff/0x256 [ 3088.468903] ? do_raw_spin_unlock+0x75/0x190 [ 3088.474977] ? _raw_spin_unlock+0x12/0x30 [ 3088.480790] do_sys_openat2+0x2b4/0x410 [ 3088.485138] do_sys_open+0x73/0xa0 [ 3088.486374] __x64_sys_openat+0x24/0x30 [ 3088.495714] do_syscall_64+0xc1/0x440 [ 3088.501261] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3088.508836] RIP: 0033:0x7fdc0afbf332 [ 3088.515139] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3088.538689] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3088.546891] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3088.560247] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3088.572415] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3088.584428] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3088.593485] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3092.049489] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3092.054825] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3092.064173] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 3092.067546] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3092.067858] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3092.067867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3092.067870] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3092.068102] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3092.068123] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3092.068349] RAX: 000000000000360a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3092.068351] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3092.068353] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3092.068355] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3092.068357] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3092.068366] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3092.068369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3092.068371] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3092.068379] Call Trace: [ 3092.068384] [ 3092.068390] ? show_regs.cold.9+0x22/0x2f [ 3092.068400] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3092.068412] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3092.068417] ? watchdog+0x40/0x40 [ 3092.068421] ? hrtimer_interrupt+0x128/0x2c0 [ 3092.068429] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3092.068445] ? apic_timer_interrupt+0xf/0x20 [ 3092.068450] [ 3092.068453] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3092.068464] do_raw_spin_lock+0x177/0x190 [ 3092.068702] _raw_spin_lock+0x21/0x30 [ 3092.068720] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3092.069047] ? do_raw_spin_unlock+0x75/0x190 [ 3092.069052] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3092.069282] ? do_wait_intr+0xf0/0xf0 [ 3092.069291] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3092.111672] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3092.111734] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3092.111742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3092.111744] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3092.111758] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 3092.111763] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3092.111767] RAX: 0000000000006019 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3092.111769] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3092.111772] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3092.111774] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3092.111776] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3092.111779] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3092.111782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3092.111784] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3092.111794] Call Trace: [ 3092.111800] [ 3092.111805] ? show_regs.cold.9+0x22/0x2f [ 3092.111816] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3092.111822] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3092.111827] ? watchdog+0x40/0x40 [ 3092.111831] ? hrtimer_interrupt+0x128/0x2c0 [ 3092.111836] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3092.158226] mdc_close+0x53b/0xd70 [mdc] [ 3092.176205] ? apic_timer_interrupt+0xf/0x20 [ 3092.176227] [ 3092.176232] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3092.176239] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3092.176242] ? dump_stack+0xf6/0x10e [ 3092.176249] do_raw_spin_lock+0x177/0x190 [ 3092.176255] _raw_spin_lock+0x21/0x30 [ 3092.176262] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3092.176531] obd_mod_free+0x46/0x140 [ptlrpc] [ 3092.176740] mdc_commit_open+0x91/0xb0 [mdc] [ 3092.176779] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3092.176977] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3092.177180] after_reply+0xe2c/0x1780 [ptlrpc] [ 3092.177376] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3092.177577] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3092.181836] lmv_close+0x18d/0x490 [lmv] [ 3092.181870] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3092.181983] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3092.182059] ll_md_close+0x274/0x800 [lustre] [ 3092.182141] ll_file_release+0x128/0xc50 [lustre] [ 3092.182218] __fput+0x10d/0x440 [ 3092.182228] ____fput+0x12/0x20 [ 3092.182233] task_work_run+0xc6/0x110 [ 3092.182240] exit_to_usermode_loop+0x1dd/0x1f0 [ 3092.182246] do_syscall_64+0x3d6/0x440 [ 3092.182251] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3092.182259] RIP: 0033:0x7fdc0afbfcd8 [ 3092.182266] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3092.182270] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3092.182275] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3092.182278] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3092.182280] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3092.182282] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3092.182284] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3092.772370] ? do_wait_intr+0xf0/0xf0 [ 3092.774396] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3092.782766] kthread+0x1d1/0x200 [ 3092.784867] ? set_kthread_struct+0x70/0x70 [ 3092.791061] ret_from_fork+0x1f/0x30 [ 3097.568931] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1072s! [ 3097.573675] Showing busy workqueues and worker pools: [ 3097.577173] workqueue events: flags=0x0 [ 3097.578753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3097.589754] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3097.596806] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3097.600969] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3097.607901] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3097.611652] pending: kfree_rcu_monitor [ 3097.615702] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3097.618873] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3097.628119] workqueue events_long: flags=0x0 [ 3097.631648] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3097.635098] pending: lu_objects_flush [obdclass] [ 3097.638582] workqueue events_unbound: flags=0x2 [ 3097.640229] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3097.643926] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3097.652855] workqueue events_power_efficient: flags=0x80 [ 3097.656785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3097.659580] pending: neigh_periodic_work, check_lifetime [ 3097.662196] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3097.670030] pending: do_cache_clean, neigh_periodic_work [ 3097.672102] workqueue mm_percpu_wq: flags=0x8 [ 3097.673556] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3097.676055] pending: lru_add_drain_per_cpu [ 3097.678060] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3097.680298] pending: vmstat_update, lru_add_drain_per_cpu [ 3097.682159] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3097.684748] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3097.689701] workqueue writeback: flags=0x4a [ 3097.691351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3097.693961] pending: wb_workfn, wb_workfn [ 3097.696053] workqueue xprtiod: flags=0xa [ 3097.697323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3097.701925] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3097.706743] workqueue ipv6_addrconf: flags=0x40008 [ 3097.711475] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3097.717447] pending: addrconf_verify_work [ 3097.720349] workqueue knbd0-recv: flags=0x1a [ 3097.722922] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3097.726148] in-flight: 655:recv_work [ 3097.728198] workqueue ptlrpc_pinger: flags=0x2 [ 3097.730701] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3097.736097] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3097.741296] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1072s workers=4 idle: 36396 38 11188 [ 3097.746760] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3088s workers=2 idle: 658 [ 3108.053822] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3108.058559] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3108.099786] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3108.109701] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3108.114860] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3108.120913] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3108.127592] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3108.147634] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3108.153490] RAX: 0000000000006a85 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3108.160529] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3108.165572] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3108.171718] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3108.177376] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3108.184459] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3108.190670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3108.195970] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3108.198830] Call Trace: [ 3108.199834] [ 3108.200652] ? show_regs.cold.9+0x22/0x2f [ 3108.202294] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3108.203938] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3108.205292] ? watchdog+0x40/0x40 [ 3108.207303] ? hrtimer_interrupt+0x128/0x2c0 [ 3108.208769] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3108.211210] ? apic_timer_interrupt+0xf/0x20 [ 3108.215332] [ 3108.216021] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3108.219115] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3108.222919] do_raw_spin_lock+0x177/0x190 [ 3108.226822] _raw_spin_lock+0x21/0x30 [ 3108.228755] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3108.230985] process_one_work+0x2c8/0x700 [ 3108.232616] worker_thread+0x296/0x6e0 [ 3108.234393] ? rescuer_thread+0x570/0x570 [ 3108.238856] kthread+0x1d1/0x200 [ 3108.239754] ? set_kthread_struct+0x70/0x70 [ 3108.243679] ret_from_fork+0x1f/0x30 [ 3116.054265] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 3116.058773] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3116.104577] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3116.121456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3116.127585] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3116.144053] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3116.177016] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3116.181379] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3116.185280] RDX: 000000000000647c RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3116.189020] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3116.192493] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3116.195738] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3116.199409] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3116.203313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3116.206466] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3116.209661] Call Trace: [ 3116.210562] [ 3116.211703] ? show_regs.cold.9+0x22/0x2f [ 3116.213639] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3116.216201] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3116.223052] ? watchdog+0x40/0x40 [ 3116.226080] ? hrtimer_interrupt+0x128/0x2c0 [ 3116.229559] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3116.238440] ? apic_timer_interrupt+0xf/0x20 [ 3116.242402] [ 3116.242962] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3116.248532] do_raw_spin_lock+0x177/0x190 [ 3116.249930] _raw_spin_lock+0x21/0x30 [ 3116.258804] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3116.270960] ? do_raw_spin_unlock+0x75/0x190 [ 3116.280099] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3116.292998] ? do_wait_intr+0xf0/0xf0 [ 3116.302561] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3116.318066] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3116.327662] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3116.338752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3116.349751] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3116.362293] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3116.371785] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3116.373275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3116.386661] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3116.388771] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3116.401018] lmv_intent_open+0x338/0x1150 [lmv] [ 3116.408632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3116.413066] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3116.420753] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3116.435743] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3116.437726] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3116.444562] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3116.455756] ? do_raw_spin_unlock+0x75/0x190 [ 3116.458576] ? _raw_spin_unlock+0x12/0x30 [ 3116.463496] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3116.469326] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3116.471246] ll_atomic_open+0x36a/0xd70 [lustre] [ 3116.473773] lookup_open+0x356/0xb40 [ 3116.480271] do_last+0x313/0xfc0 [ 3116.487131] ? nd_jump_root+0xe5/0x160 [ 3116.489214] ? path_init+0x437/0x520 [ 3116.494219] path_openat+0xf7/0x500 [ 3116.495530] ? do_raw_spin_unlock+0x75/0x190 [ 3116.504502] do_filp_open+0x99/0x140 [ 3116.505848] ? getname_flags+0x6e/0x330 [ 3116.511742] ? __check_object_size+0xff/0x256 [ 3116.516301] ? do_raw_spin_unlock+0x75/0x190 [ 3116.522067] ? _raw_spin_unlock+0x12/0x30 [ 3116.524750] do_sys_openat2+0x2b4/0x410 [ 3116.528407] do_sys_open+0x73/0xa0 [ 3116.529704] __x64_sys_openat+0x24/0x30 [ 3116.530744] do_syscall_64+0xc1/0x440 [ 3116.543684] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3116.548819] RIP: 0033:0x7fdc0afbf332 [ 3116.558325] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3116.588043] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3116.591455] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3116.607566] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3116.616532] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3116.625643] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3116.647233] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3120.055591] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3120.063232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3120.105299] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3120.111630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3120.127068] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3120.130643] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3120.144847] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3120.159985] RAX: 0000000000007d1d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3120.166532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3120.170329] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3120.177110] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3120.182146] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3120.190588] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3120.195636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3120.199179] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3120.202759] Call Trace: [ 3120.204983] [ 3120.206734] ? show_regs.cold.9+0x22/0x2f [ 3120.215125] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3120.216918] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3120.237119] ? watchdog+0x40/0x40 [ 3120.238324] ? hrtimer_interrupt+0x128/0x2c0 [ 3120.239799] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3120.245592] ? apic_timer_interrupt+0xf/0x20 [ 3120.248228] [ 3120.249771] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3120.256454] do_raw_spin_lock+0x177/0x190 [ 3120.264894] _raw_spin_lock+0x21/0x30 [ 3120.274063] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3120.276281] ? do_raw_spin_unlock+0x75/0x190 [ 3120.280049] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3120.284792] ? do_wait_intr+0xf0/0xf0 [ 3120.288411] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3120.298156] mdc_close+0x53b/0xd70 [mdc] [ 3120.300501] lmv_close+0x18d/0x490 [lmv] [ 3120.305558] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3120.309405] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3120.312374] ll_md_close+0x274/0x800 [lustre] [ 3120.314788] ll_file_release+0x128/0xc50 [lustre] [ 3120.316579] __fput+0x10d/0x440 [ 3120.317697] ____fput+0x12/0x20 [ 3120.322419] task_work_run+0xc6/0x110 [ 3120.325390] exit_to_usermode_loop+0x1dd/0x1f0 [ 3120.331430] do_syscall_64+0x3d6/0x440 [ 3120.333569] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3120.339187] RIP: 0033:0x7fdc0afbfcd8 [ 3120.342746] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3120.359767] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3120.365561] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3120.376893] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3120.390576] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3120.402590] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3120.406795] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3124.049745] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3124.056057] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3124.104325] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3124.116630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3124.124331] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3124.129972] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3124.145391] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3124.152670] RAX: 0000000000002a0d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3124.159221] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3124.165011] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3124.169622] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3124.177112] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3124.183661] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3124.189274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3124.190835] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3124.201035] Call Trace: [ 3124.205653] [ 3124.206674] ? show_regs.cold.9+0x22/0x2f [ 3124.208850] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3124.212417] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3124.214521] ? watchdog+0x40/0x40 [ 3124.216419] ? hrtimer_interrupt+0x128/0x2c0 [ 3124.220050] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3124.225442] ? apic_timer_interrupt+0xf/0x20 [ 3124.232517] [ 3124.233391] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3124.245932] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3124.253259] ? dump_stack+0xf6/0x10e [ 3124.254211] do_raw_spin_lock+0x177/0x190 [ 3124.261656] _raw_spin_lock+0x21/0x30 [ 3124.266666] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3124.272152] obd_mod_free+0x46/0x140 [ptlrpc] [ 3124.276945] mdc_commit_open+0x91/0xb0 [mdc] [ 3124.282953] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3124.284679] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3124.292242] after_reply+0xe2c/0x1780 [ptlrpc] [ 3124.298255] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3124.302710] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3124.305136] ? do_wait_intr+0xf0/0xf0 [ 3124.310376] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3124.313188] kthread+0x1d1/0x200 [ 3124.315038] ? set_kthread_struct+0x70/0x70 [ 3124.316370] ret_from_fork+0x1f/0x30 [ 3127.784813] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1102s! [ 3127.796105] Showing busy workqueues and worker pools: [ 3127.798875] workqueue events: flags=0x0 [ 3127.801388] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3127.807618] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3127.813923] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3127.829579] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3127.840339] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3127.849420] pending: kfree_rcu_monitor [ 3127.854376] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3127.862352] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3127.874178] workqueue events_long: flags=0x0 [ 3127.878276] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3127.880615] pending: lu_objects_flush [obdclass] [ 3127.882571] workqueue events_unbound: flags=0x2 [ 3127.884437] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3127.887595] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3127.893257] workqueue events_power_efficient: flags=0x80 [ 3127.895298] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3127.900722] pending: neigh_periodic_work, check_lifetime [ 3127.903153] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3127.905434] pending: do_cache_clean, neigh_periodic_work [ 3127.921963] workqueue mm_percpu_wq: flags=0x8 [ 3127.930858] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3127.938968] pending: lru_add_drain_per_cpu [ 3127.941433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3127.947534] pending: vmstat_update, lru_add_drain_per_cpu [ 3127.955149] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3127.960620] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3127.967914] workqueue writeback: flags=0x4a [ 3127.969478] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3127.974147] pending: wb_workfn, wb_workfn [ 3127.976050] workqueue xprtiod: flags=0xa [ 3127.977153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3127.979455] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3127.982654] workqueue ipv6_addrconf: flags=0x40008 [ 3127.984600] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3127.991157] pending: addrconf_verify_work [ 3127.997127] workqueue knbd0-recv: flags=0x1a [ 3128.001875] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3128.012031] in-flight: 655:recv_work [ 3128.017182] workqueue ptlrpc_pinger: flags=0x2 [ 3128.024012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3128.031400] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3128.039431] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1103s workers=4 idle: 36396 38 11188 [ 3128.042374] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3118s workers=2 idle: 658 [ 3136.051130] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3136.053434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3136.066791] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3136.070525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3136.073626] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3136.075665] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3136.077314] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3136.083764] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3136.086297] RAX: 0000000000003467 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3136.089188] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3136.091743] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3136.093841] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3136.095747] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3136.097816] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3136.100247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3136.102149] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3136.104311] Call Trace: [ 3136.105123] [ 3136.105734] ? show_regs.cold.9+0x22/0x2f [ 3136.106989] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3136.108315] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3136.109689] ? watchdog+0x40/0x40 [ 3136.110673] ? hrtimer_interrupt+0x128/0x2c0 [ 3136.112122] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3136.113765] ? apic_timer_interrupt+0xf/0x20 [ 3136.115275] [ 3136.116065] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3136.118045] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3136.120108] do_raw_spin_lock+0x177/0x190 [ 3136.121524] _raw_spin_lock+0x21/0x30 [ 3136.122926] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3136.125037] process_one_work+0x2c8/0x700 [ 3136.126373] worker_thread+0x296/0x6e0 [ 3136.127450] ? rescuer_thread+0x570/0x570 [ 3136.128991] kthread+0x1d1/0x200 [ 3136.130247] ? set_kthread_struct+0x70/0x70 [ 3136.131761] ret_from_fork+0x1f/0x30 [ 3144.052314] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 3144.064228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3144.109762] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3144.122618] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3144.134019] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3144.136208] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3144.157099] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3144.169137] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3144.176004] RDX: 00000000000045be RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3144.183863] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3144.192566] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3144.195147] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3144.201058] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3144.212935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3144.228683] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3144.244238] Call Trace: [ 3144.250681] [ 3144.255804] ? show_regs.cold.9+0x22/0x2f [ 3144.260236] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3144.268762] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3144.270885] ? watchdog+0x40/0x40 [ 3144.279681] ? hrtimer_interrupt+0x128/0x2c0 [ 3144.280977] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3144.293477] ? apic_timer_interrupt+0xf/0x20 [ 3144.296956] [ 3144.297744] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3144.309257] do_raw_spin_lock+0x177/0x190 [ 3144.323016] _raw_spin_lock+0x21/0x30 [ 3144.327782] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3144.338296] ? do_raw_spin_unlock+0x75/0x190 [ 3144.347494] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3144.350455] ? do_wait_intr+0xf0/0xf0 [ 3144.355591] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3144.366998] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3144.381540] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3144.400621] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3144.402662] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3144.417831] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3144.420573] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3144.424354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3144.427643] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3144.439503] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3144.443913] lmv_intent_open+0x338/0x1150 [lmv] [ 3144.445645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3144.448684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3144.450734] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3144.453837] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3144.456223] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3144.459072] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3144.461487] ? do_raw_spin_unlock+0x75/0x190 [ 3144.463727] ? _raw_spin_unlock+0x12/0x30 [ 3144.465811] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3144.468354] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3144.471161] ll_atomic_open+0x36a/0xd70 [lustre] [ 3144.473528] lookup_open+0x356/0xb40 [ 3144.475164] do_last+0x313/0xfc0 [ 3144.476661] ? nd_jump_root+0xe5/0x160 [ 3144.478371] ? path_init+0x437/0x520 [ 3144.480129] path_openat+0xf7/0x500 [ 3144.481702] ? do_raw_spin_unlock+0x75/0x190 [ 3144.483644] do_filp_open+0x99/0x140 [ 3144.485150] ? getname_flags+0x6e/0x330 [ 3144.486242] ? __check_object_size+0xff/0x256 [ 3144.487576] ? do_raw_spin_unlock+0x75/0x190 [ 3144.489307] ? _raw_spin_unlock+0x12/0x30 [ 3144.491139] do_sys_openat2+0x2b4/0x410 [ 3144.492768] do_sys_open+0x73/0xa0 [ 3144.494357] __x64_sys_openat+0x24/0x30 [ 3144.496131] do_syscall_64+0xc1/0x440 [ 3144.497806] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3144.500135] RIP: 0033:0x7fdc0afbf332 [ 3144.501762] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3144.510210] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3144.513700] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3144.516917] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3144.519780] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3144.523078] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3144.526493] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3148.055631] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3148.067695] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3148.101746] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3148.109872] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3148.113667] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3148.119857] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3148.145702] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3148.149530] RAX: 000000000000289a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3148.153187] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3148.156914] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3148.160058] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3148.165066] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3148.168648] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3148.172497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3148.174868] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3148.177467] Call Trace: [ 3148.178438] [ 3148.179176] ? show_regs.cold.9+0x22/0x2f [ 3148.180930] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3148.182674] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3148.184460] ? watchdog+0x40/0x40 [ 3148.185712] ? hrtimer_interrupt+0x128/0x2c0 [ 3148.187090] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3148.196275] ? apic_timer_interrupt+0xf/0x20 [ 3148.199259] [ 3148.202068] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3148.204614] do_raw_spin_lock+0x177/0x190 [ 3148.207711] _raw_spin_lock+0x21/0x30 [ 3148.209391] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3148.213101] ? do_raw_spin_unlock+0x75/0x190 [ 3148.216278] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3148.221951] ? do_wait_intr+0xf0/0xf0 [ 3148.228097] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3148.236576] mdc_close+0x53b/0xd70 [mdc] [ 3148.244067] lmv_close+0x18d/0x490 [lmv] [ 3148.249273] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3148.258622] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3148.263242] ll_md_close+0x274/0x800 [lustre] [ 3148.266994] ll_file_release+0x128/0xc50 [lustre] [ 3148.270231] __fput+0x10d/0x440 [ 3148.272756] ____fput+0x12/0x20 [ 3148.274395] task_work_run+0xc6/0x110 [ 3148.280928] exit_to_usermode_loop+0x1dd/0x1f0 [ 3148.282067] do_syscall_64+0x3d6/0x440 [ 3148.283072] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3148.293607] RIP: 0033:0x7fdc0afbfcd8 [ 3148.297599] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3148.318106] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3148.330716] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3148.335940] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3148.341801] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3148.351221] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3148.359242] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3156.048082] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 3156.051896] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3156.089483] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3156.101448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3156.115924] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3156.121283] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3156.144140] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3156.150889] RAX: 0000000000005457 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3156.158319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3156.162879] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3156.166562] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3156.176466] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3156.180631] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3156.189801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3156.192707] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3156.196294] Call Trace: [ 3156.197548] [ 3156.198641] ? show_regs.cold.9+0x22/0x2f [ 3156.200311] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3156.202604] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3156.207832] ? watchdog+0x40/0x40 [ 3156.209725] ? hrtimer_interrupt+0x128/0x2c0 [ 3156.211773] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3156.214504] ? apic_timer_interrupt+0xf/0x20 [ 3156.216725] [ 3156.217537] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3156.220368] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3156.222369] ? dump_stack+0xf6/0x10e [ 3156.225558] do_raw_spin_lock+0x177/0x190 [ 3156.227116] _raw_spin_lock+0x21/0x30 [ 3156.229733] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3156.232678] obd_mod_free+0x46/0x140 [ptlrpc] [ 3156.237421] mdc_commit_open+0x91/0xb0 [mdc] [ 3156.242803] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3156.249807] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3156.257677] after_reply+0xe2c/0x1780 [ptlrpc] [ 3156.261961] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3156.266104] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3156.268911] ? do_wait_intr+0xf0/0xf0 [ 3156.270457] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3156.272829] kthread+0x1d1/0x200 [ 3156.277670] ? set_kthread_struct+0x70/0x70 [ 3156.280151] ret_from_fork+0x1f/0x30 [ 3158.495048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1133s! [ 3158.503503] Showing busy workqueues and worker pools: [ 3158.510397] workqueue events: flags=0x0 [ 3158.515581] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3158.523130] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3158.528977] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3158.536202] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3158.547674] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3158.555058] pending: kfree_rcu_monitor [ 3158.563563] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3158.573423] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3158.583585] workqueue events_long: flags=0x0 [ 3158.587306] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3158.594913] pending: lu_objects_flush [obdclass] [ 3158.602217] workqueue events_unbound: flags=0x2 [ 3158.604513] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3158.611716] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3158.624634] workqueue events_power_efficient: flags=0x80 [ 3158.632130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3158.654143] pending: neigh_periodic_work, check_lifetime [ 3158.668357] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3158.681303] pending: do_cache_clean, neigh_periodic_work [ 3158.693732] workqueue mm_percpu_wq: flags=0x8 [ 3158.700016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3158.706812] pending: lru_add_drain_per_cpu [ 3158.712214] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3158.721399] pending: vmstat_update, lru_add_drain_per_cpu [ 3158.729040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3158.740851] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3158.750739] workqueue writeback: flags=0x4a [ 3158.754835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3158.759410] pending: wb_workfn, wb_workfn [ 3158.767146] workqueue xprtiod: flags=0xa [ 3158.769553] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3158.773060] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3158.777398] workqueue ipv6_addrconf: flags=0x40008 [ 3158.780251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3158.783839] pending: addrconf_verify_work [ 3158.786324] workqueue knbd0-recv: flags=0x1a [ 3158.788388] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3158.791426] in-flight: 655:recv_work [ 3158.793366] workqueue ptlrpc_pinger: flags=0x2 [ 3158.795305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3158.797248] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3158.800018] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1133s workers=4 idle: 36396 38 11188 [ 3158.804200] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3149s workers=2 idle: 658 [ 3159.366031] rcu: INFO: rcu_sched self-detected stall on CPU [ 3159.370704] rcu: 0-...!: (669855 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 3159.375271] (t=1140023 jiffies g=270761 q=2924) [ 3159.377693] rcu: rcu_sched kthread starved for 1134955 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 3159.382778] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3159.386863] rcu: RCU grace-period kthread stack dump: [ 3159.389514] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3159.394505] Call Trace: [ 3159.395798] __schedule+0x351/0xcb0 [ 3159.397462] schedule+0xc0/0x180 [ 3159.401963] schedule_timeout+0xb4/0x190 [ 3159.404248] ? do_raw_spin_unlock+0x75/0x190 [ 3159.411098] ? __next_timer_interrupt+0x160/0x160 [ 3159.413069] ? prepare_to_swait_event+0xf3/0x140 [ 3159.416323] rcu_gp_kthread+0x8dd/0x15b0 [ 3159.418082] ? force_qs_rnp+0x2c0/0x2c0 [ 3159.419897] kthread+0x1d1/0x200 [ 3159.420991] ? set_kthread_struct+0x70/0x70 [ 3159.426509] ret_from_fork+0x1f/0x30 [ 3159.429575] rcu: Stack dump where RCU GP kthread last ran: [ 3159.433959] Sending NMI from CPU 0 to CPUs 3: [ 3159.442578] NMI backtrace for cpu 3 [ 3159.442585] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3159.442587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3159.442589] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3159.442591] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3159.442593] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 [ 3159.442596] RAX: 0000000000006703 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3159.442597] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3159.442599] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3159.442600] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3159.442602] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3159.442603] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3159.442605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3159.442606] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3159.442607] Call Trace: [ 3159.442608] [ 3159.442609] ? show_regs.cold.9+0x22/0x2f [ 3159.442610] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3159.442611] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3159.442612] ? nmi_handle+0xbd/0x210 [ 3159.442614] ? default_do_nmi+0x4b/0x1b0 [ 3159.442615] ? do_nmi+0x280/0x380 [ 3159.442616] ? end_repeat_nmi+0x16/0x1f [ 3159.442617] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3159.442618] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3159.442620] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3159.442621] [ 3159.442622] do_raw_spin_lock+0x177/0x190 [ 3159.442623] _raw_spin_lock+0x21/0x30 [ 3159.442624] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3159.442625] ? do_raw_spin_unlock+0x75/0x190 [ 3159.442626] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3159.442627] ? do_wait_intr+0xf0/0xf0 [ 3159.442628] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3159.442629] mdc_close+0x53b/0xd70 [mdc] [ 3159.442630] lmv_close+0x18d/0x490 [lmv] [ 3159.442631] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3159.442632] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3159.442633] ll_md_close+0x274/0x800 [lustre] [ 3159.442634] ll_file_release+0x128/0xc50 [lustre] [ 3159.442635] __fput+0x10d/0x440 [ 3159.442636] ____fput+0x12/0x20 [ 3159.442637] task_work_run+0xc6/0x110 [ 3159.442638] exit_to_usermode_loop+0x1dd/0x1f0 [ 3159.442639] do_syscall_64+0x3d6/0x440 [ 3159.442640] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3159.442641] RIP: 0033:0x7fdc0afbfcd8 [ 3159.442643] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3159.442645] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3159.442648] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3159.442650] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3159.442651] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3159.442653] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3159.442654] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3159.444353] NMI backtrace for cpu 0 [ 3159.647800] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3159.654698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3159.659927] Call Trace: [ 3159.661362] [ 3159.662788] ? dump_stack+0xbb/0x10e [ 3159.664840] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3159.666903] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3159.669089] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3159.670869] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3159.675697] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3159.677016] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3159.678069] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3159.681592] ? update_process_times+0x6d/0xc0 [ 3159.690072] ? tick_sched_handle.isra.18+0x27/0x90 [ 3159.693591] ? tick_sched_timer+0x53/0xb0 [ 3159.696386] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3159.699637] ? tick_sched_handle.isra.18+0x90/0x90 [ 3159.706981] ? hrtimer_interrupt+0x128/0x2c0 [ 3159.709455] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3159.713378] ? apic_timer_interrupt+0xf/0x20 [ 3159.717720] [ 3159.718965] ? native_safe_halt+0x2e/0x50 [ 3159.721837] ? kvm_wait+0xcd/0xe0 [ 3159.723997] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3159.727603] ? dump_stack+0xf6/0x10e [ 3159.732555] ? do_raw_spin_lock+0x177/0x190 [ 3159.733994] ? _raw_spin_lock+0x21/0x30 [ 3159.736645] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3159.741588] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3159.745574] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3159.747815] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3159.751333] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3159.757904] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3159.760887] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3159.763869] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3159.766211] ? do_wait_intr+0xf0/0xf0 [ 3159.768109] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3159.770261] ? kthread+0x1d1/0x200 [ 3159.771561] ? set_kthread_struct+0x70/0x70 [ 3159.772964] ? ret_from_fork+0x1f/0x30 [ 3164.051640] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3164.062174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3164.126474] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3164.146371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3164.155759] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3164.161817] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3164.173346] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3164.195427] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3164.202551] RAX: 0000000000007f90 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3164.210423] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3164.214807] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3164.218299] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3164.223908] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3164.229195] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3164.236462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3164.241019] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3164.247920] Call Trace: [ 3164.249150] [ 3164.250156] ? show_regs.cold.9+0x22/0x2f [ 3164.253627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3164.256801] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3164.258207] ? watchdog+0x40/0x40 [ 3164.259225] ? hrtimer_interrupt+0x128/0x2c0 [ 3164.267924] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3164.271811] ? apic_timer_interrupt+0xf/0x20 [ 3164.276402] [ 3164.277402] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3164.280985] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3164.283699] do_raw_spin_lock+0x177/0x190 [ 3164.285947] _raw_spin_lock+0x21/0x30 [ 3164.287963] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3164.291718] process_one_work+0x2c8/0x700 [ 3164.293792] worker_thread+0x296/0x6e0 [ 3164.298410] ? rescuer_thread+0x570/0x570 [ 3164.302739] kthread+0x1d1/0x200 [ 3164.306803] ? set_kthread_struct+0x70/0x70 [ 3164.309439] ret_from_fork+0x1f/0x30 [ 3172.052015] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 3172.060982] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3172.097984] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3172.108135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3172.120956] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3172.128038] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3172.158356] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3172.163126] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3172.173552] RDX: 0000000000007bbd RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3172.181103] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3172.188379] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3172.191141] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3172.194998] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3172.204868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3172.209232] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3172.214408] Call Trace: [ 3172.221435] [ 3172.222353] ? show_regs.cold.9+0x22/0x2f [ 3172.226367] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3172.229312] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3172.232960] ? watchdog+0x40/0x40 [ 3172.234448] ? hrtimer_interrupt+0x128/0x2c0 [ 3172.235901] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3172.237703] ? apic_timer_interrupt+0xf/0x20 [ 3172.240271] [ 3172.242095] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3172.247168] do_raw_spin_lock+0x177/0x190 [ 3172.248757] _raw_spin_lock+0x21/0x30 [ 3172.250989] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3172.254157] ? do_raw_spin_unlock+0x75/0x190 [ 3172.255721] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3172.257875] ? do_wait_intr+0xf0/0xf0 [ 3172.262188] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3172.264467] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3172.267561] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3172.272652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3172.280102] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3172.283331] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3172.287270] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3172.289665] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3172.299688] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3172.304704] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3172.306649] lmv_intent_open+0x338/0x1150 [lmv] [ 3172.310483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3172.315220] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3172.319254] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3172.321240] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3172.323426] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3172.327461] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3172.332733] ? do_raw_spin_unlock+0x75/0x190 [ 3172.336451] ? _raw_spin_unlock+0x12/0x30 [ 3172.339456] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3172.342708] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3172.348011] ll_atomic_open+0x36a/0xd70 [lustre] [ 3172.350667] lookup_open+0x356/0xb40 [ 3172.355767] do_last+0x313/0xfc0 [ 3172.364294] ? nd_jump_root+0xe5/0x160 [ 3172.365624] ? path_init+0x437/0x520 [ 3172.366932] path_openat+0xf7/0x500 [ 3172.373257] ? do_raw_spin_unlock+0x75/0x190 [ 3172.379478] do_filp_open+0x99/0x140 [ 3172.381779] ? getname_flags+0x6e/0x330 [ 3172.385805] ? __check_object_size+0xff/0x256 [ 3172.391634] ? do_raw_spin_unlock+0x75/0x190 [ 3172.397224] ? _raw_spin_unlock+0x12/0x30 [ 3172.403082] do_sys_openat2+0x2b4/0x410 [ 3172.407652] do_sys_open+0x73/0xa0 [ 3172.410604] __x64_sys_openat+0x24/0x30 [ 3172.412378] do_syscall_64+0xc1/0x440 [ 3172.414074] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3172.416175] RIP: 0033:0x7fdc0afbf332 [ 3172.417625] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3172.426182] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3172.431844] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3172.439873] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3172.441426] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3172.444712] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3172.450505] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3184.050723] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3184.055110] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3184.057009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3184.105150] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3184.105165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3184.105168] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3184.105301] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3184.105306] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3184.105312] RAX: 0000000000007e93 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3184.105314] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3184.105317] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3184.105319] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3184.105322] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3184.105325] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3184.105328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3184.105330] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3184.105339] Call Trace: [ 3184.105344] [ 3184.105351] ? show_regs.cold.9+0x22/0x2f [ 3184.105362] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3184.105368] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3184.105373] ? watchdog+0x40/0x40 [ 3184.105377] ? hrtimer_interrupt+0x128/0x2c0 [ 3184.105381] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3184.105406] ? apic_timer_interrupt+0xf/0x20 [ 3184.105410] [ 3184.105413] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3184.105418] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3184.105421] ? dump_stack+0xf6/0x10e [ 3184.105426] do_raw_spin_lock+0x177/0x190 [ 3184.105431] _raw_spin_lock+0x21/0x30 [ 3184.105438] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3184.105895] obd_mod_free+0x46/0x140 [ptlrpc] [ 3184.106087] mdc_commit_open+0x91/0xb0 [mdc] [ 3184.106138] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3184.112175] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3184.114956] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3184.114964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3184.114967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3184.114982] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3184.114986] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3184.114990] RAX: 0000000000005a0e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3184.114992] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3184.114994] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3184.114996] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3184.114998] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3184.115002] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3184.115004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3184.115006] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3184.115015] Call Trace: [ 3184.115020] [ 3184.115025] ? show_regs.cold.9+0x22/0x2f [ 3184.115034] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3184.115040] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3184.115044] ? watchdog+0x40/0x40 [ 3184.115048] ? hrtimer_interrupt+0x128/0x2c0 [ 3184.115051] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3184.115059] ? apic_timer_interrupt+0xf/0x20 [ 3184.115063] [ 3184.115066] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3184.115072] do_raw_spin_lock+0x177/0x190 [ 3184.115076] _raw_spin_lock+0x21/0x30 [ 3184.115081] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3184.115415] ? do_raw_spin_unlock+0x75/0x190 [ 3184.115422] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3184.115851] ? do_wait_intr+0xf0/0xf0 [ 3184.115859] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3184.116049] mdc_close+0x53b/0xd70 [mdc] [ 3184.116089] lmv_close+0x18d/0x490 [lmv] [ 3184.116130] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3184.116371] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3184.116618] ll_md_close+0x274/0x800 [lustre] [ 3184.116771] ll_file_release+0x128/0xc50 [lustre] [ 3184.116851] __fput+0x10d/0x440 [ 3184.116861] ____fput+0x12/0x20 [ 3184.116865] task_work_run+0xc6/0x110 [ 3184.116872] exit_to_usermode_loop+0x1dd/0x1f0 [ 3184.116879] do_syscall_64+0x3d6/0x440 [ 3184.116883] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3184.116890] RIP: 0033:0x7fdc0afbfcd8 [ 3184.116897] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3184.116900] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3184.116905] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3184.116907] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3184.116909] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3184.116910] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3184.116912] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3184.425809] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3184.428637] after_reply+0xe2c/0x1780 [ptlrpc] [ 3184.431776] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3184.434432] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3184.436863] ? do_wait_intr+0xf0/0xf0 [ 3184.438553] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3184.441072] kthread+0x1d1/0x200 [ 3184.443300] ? set_kthread_struct+0x70/0x70 [ 3184.444905] ret_from_fork+0x1f/0x30 [ 3189.221976] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1164s! [ 3189.243587] Showing busy workqueues and worker pools: [ 3189.254446] workqueue events: flags=0x0 [ 3189.260225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3189.267461] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3189.278693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3189.295640] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3189.317069] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3189.326402] pending: kfree_rcu_monitor [ 3189.331262] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3189.334616] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3189.347838] workqueue events_long: flags=0x0 [ 3189.353236] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3189.360886] pending: lu_objects_flush [obdclass] [ 3189.368348] workqueue events_unbound: flags=0x2 [ 3189.370196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3189.375488] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3189.391177] workqueue events_power_efficient: flags=0x80 [ 3189.397413] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3189.410692] pending: neigh_periodic_work, check_lifetime [ 3189.423134] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3189.431871] pending: do_cache_clean, neigh_periodic_work [ 3189.437141] workqueue mm_percpu_wq: flags=0x8 [ 3189.442947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3189.454404] pending: lru_add_drain_per_cpu [ 3189.458793] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3189.473039] pending: vmstat_update, lru_add_drain_per_cpu [ 3189.476492] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3189.497404] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3189.507029] workqueue writeback: flags=0x4a [ 3189.514940] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3189.525331] pending: wb_workfn, wb_workfn [ 3189.530363] workqueue xprtiod: flags=0xa [ 3189.540660] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3189.542876] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3189.555775] workqueue ipv6_addrconf: flags=0x40008 [ 3189.559762] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3189.563544] pending: addrconf_verify_work [ 3189.570578] workqueue knbd0-recv: flags=0x1a [ 3189.574483] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3189.581994] in-flight: 655:recv_work [ 3189.588275] workqueue ptlrpc_pinger: flags=0x2 [ 3189.593308] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3189.597975] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3189.603225] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1164s workers=4 idle: 36396 38 11188 [ 3189.608231] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3180s workers=2 idle: 658 [ 3192.054580] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3192.058269] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3192.111272] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3192.117441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3192.131482] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3192.138943] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3192.143244] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3192.165078] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3192.171350] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3192.176758] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3192.180375] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3192.187449] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3192.202081] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3192.208451] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3192.216079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3192.225019] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3192.236455] Call Trace: [ 3192.240911] [ 3192.242292] ? show_regs.cold.9+0x22/0x2f [ 3192.244524] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3192.248165] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3192.252157] ? watchdog+0x40/0x40 [ 3192.253759] ? hrtimer_interrupt+0x128/0x2c0 [ 3192.261735] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3192.266962] ? apic_timer_interrupt+0xf/0x20 [ 3192.273251] [ 3192.274161] ? native_safe_halt+0x2e/0x50 [ 3192.275578] kvm_wait+0xcd/0xe0 [ 3192.276826] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3192.292075] do_raw_spin_lock+0x177/0x190 [ 3192.295948] _raw_spin_lock+0x21/0x30 [ 3192.298634] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3192.302047] process_one_work+0x2c8/0x700 [ 3192.303998] worker_thread+0x296/0x6e0 [ 3192.310990] ? rescuer_thread+0x570/0x570 [ 3192.315062] kthread+0x1d1/0x200 [ 3192.318679] ? set_kthread_struct+0x70/0x70 [ 3192.321051] ret_from_fork+0x1f/0x30 [ 3200.056608] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 3200.063440] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3200.105186] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3200.115949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3200.126083] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3200.128966] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3200.152104] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3200.160240] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3200.176498] RDX: 0000000000007e10 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3200.189047] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3200.197108] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3200.203657] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3200.214402] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3200.222864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3200.224981] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3200.233033] Call Trace: [ 3200.235011] [ 3200.236257] ? show_regs.cold.9+0x22/0x2f [ 3200.238179] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3200.241376] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3200.244518] ? watchdog+0x40/0x40 [ 3200.246192] ? hrtimer_interrupt+0x128/0x2c0 [ 3200.251936] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3200.256967] ? apic_timer_interrupt+0xf/0x20 [ 3200.259128] [ 3200.259741] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3200.264679] do_raw_spin_lock+0x177/0x190 [ 3200.269933] _raw_spin_lock+0x21/0x30 [ 3200.274551] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3200.280840] ? do_raw_spin_unlock+0x75/0x190 [ 3200.283882] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3200.286479] ? do_wait_intr+0xf0/0xf0 [ 3200.288646] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3200.291339] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3200.294167] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3200.297114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3200.300069] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3200.302534] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3200.304651] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3200.306574] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3200.308988] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3200.311546] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3200.314172] lmv_intent_open+0x338/0x1150 [lmv] [ 3200.316359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3200.318254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3200.320901] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3200.323093] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3200.325490] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3200.327992] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3200.330529] ? do_raw_spin_unlock+0x75/0x190 [ 3200.332488] ? _raw_spin_unlock+0x12/0x30 [ 3200.334305] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3200.336610] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3200.338614] ll_atomic_open+0x36a/0xd70 [lustre] [ 3200.340855] lookup_open+0x356/0xb40 [ 3200.342543] do_last+0x313/0xfc0 [ 3200.344144] ? nd_jump_root+0xe5/0x160 [ 3200.345892] ? path_init+0x437/0x520 [ 3200.347600] path_openat+0xf7/0x500 [ 3200.349307] ? do_raw_spin_unlock+0x75/0x190 [ 3200.351381] do_filp_open+0x99/0x140 [ 3200.353144] ? getname_flags+0x6e/0x330 [ 3200.355060] ? __check_object_size+0xff/0x256 [ 3200.357079] ? do_raw_spin_unlock+0x75/0x190 [ 3200.359139] ? _raw_spin_unlock+0x12/0x30 [ 3200.361533] do_sys_openat2+0x2b4/0x410 [ 3200.363371] do_sys_open+0x73/0xa0 [ 3200.364976] __x64_sys_openat+0x24/0x30 [ 3200.366898] do_syscall_64+0xc1/0x440 [ 3200.368571] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3200.370741] RIP: 0033:0x7fdc0afbf332 [ 3200.372057] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3200.379377] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3200.383039] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3200.387111] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3200.390369] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3200.405339] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3200.407992] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3212.055830] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3212.071585] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3212.154032] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3212.174750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3212.196478] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3212.207447] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3212.244797] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3212.247424] RAX: 0000000000006884 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3212.262974] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3212.271182] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3212.282378] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3212.295090] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3212.311809] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3212.324306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3212.334384] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3212.343780] Call Trace: [ 3212.351530] [ 3212.352343] ? show_regs.cold.9+0x22/0x2f [ 3212.353807] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3212.365191] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3212.374221] ? watchdog+0x40/0x40 [ 3212.375732] ? hrtimer_interrupt+0x128/0x2c0 [ 3212.383427] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3212.385231] ? apic_timer_interrupt+0xf/0x20 [ 3212.394404] [ 3212.396566] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3212.402439] do_raw_spin_lock+0x177/0x190 [ 3212.403940] _raw_spin_lock+0x21/0x30 [ 3212.411826] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3212.420134] ? do_raw_spin_unlock+0x75/0x190 [ 3212.424885] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3212.438060] ? do_wait_intr+0xf0/0xf0 [ 3212.439264] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3212.443593] mdc_close+0x53b/0xd70 [mdc] [ 3212.447813] lmv_close+0x18d/0x490 [lmv] [ 3212.452756] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3212.456241] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3212.459210] ll_md_close+0x274/0x800 [lustre] [ 3212.464987] ll_file_release+0x128/0xc50 [lustre] [ 3212.469057] __fput+0x10d/0x440 [ 3212.471295] ____fput+0x12/0x20 [ 3212.477235] task_work_run+0xc6/0x110 [ 3212.486160] exit_to_usermode_loop+0x1dd/0x1f0 [ 3212.504384] do_syscall_64+0x3d6/0x440 [ 3212.505687] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3212.507474] RIP: 0033:0x7fdc0afbfcd8 [ 3212.519517] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3212.547648] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3212.550576] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3212.571649] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3212.588572] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3212.596393] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3212.603581] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3216.048601] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 3216.059242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3216.102527] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3216.110728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3216.116543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3216.119716] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3216.133572] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3216.142696] RAX: 0000000000007d8a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3216.147529] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3216.160715] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3216.167359] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3216.174643] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3216.187871] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3216.193942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3216.201577] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3216.207730] Call Trace: [ 3216.212815] [ 3216.218932] ? show_regs.cold.9+0x22/0x2f [ 3216.226432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3216.240912] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3216.254667] ? watchdog+0x40/0x40 [ 3216.263070] ? hrtimer_interrupt+0x128/0x2c0 [ 3216.264201] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3216.265576] ? apic_timer_interrupt+0xf/0x20 [ 3216.277465] [ 3216.282006] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3216.289267] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3216.293260] ? dump_stack+0xf6/0x10e [ 3216.296564] do_raw_spin_lock+0x177/0x190 [ 3216.298859] _raw_spin_lock+0x21/0x30 [ 3216.300430] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3216.310011] obd_mod_free+0x46/0x140 [ptlrpc] [ 3216.317631] mdc_commit_open+0x91/0xb0 [mdc] [ 3216.326737] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3216.332073] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3216.341857] after_reply+0xe2c/0x1780 [ptlrpc] [ 3216.343387] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3216.351429] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3216.359555] ? do_wait_intr+0xf0/0xf0 [ 3216.361734] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3216.368207] kthread+0x1d1/0x200 [ 3216.369631] ? set_kthread_struct+0x70/0x70 [ 3216.371984] ret_from_fork+0x1f/0x30 [ 3219.943235] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1195s! [ 3219.946173] Showing busy workqueues and worker pools: [ 3219.962515] workqueue events: flags=0x0 [ 3219.969316] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3219.979003] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3219.991541] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3219.995909] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3220.011310] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3220.015539] pending: kfree_rcu_monitor [ 3220.017685] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3220.021385] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3220.026019] workqueue events_long: flags=0x0 [ 3220.028281] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3220.030609] pending: lu_objects_flush [obdclass] [ 3220.032551] workqueue events_unbound: flags=0x2 [ 3220.034549] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3220.045222] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3220.051069] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3220.054288] workqueue events_power_efficient: flags=0x80 [ 3220.054301] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3220.054315] pending: neigh_periodic_work, check_lifetime [ 3220.054333] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3220.054341] pending: do_cache_clean, neigh_periodic_work [ 3220.054355] workqueue mm_percpu_wq: flags=0x8 [ 3220.054357] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3220.054362] pending: lru_add_drain_per_cpu [ 3220.054371] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3220.054378] pending: vmstat_update, lru_add_drain_per_cpu [ 3220.054387] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3220.054392] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3220.054405] workqueue writeback: flags=0x4a [ 3220.054407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3220.054414] pending: wb_workfn, wb_workfn [ 3220.054429] workqueue xprtiod: flags=0xa [ 3220.054431] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3220.054437] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3220.054456] workqueue ipv6_addrconf: flags=0x40008 [ 3220.054460] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3220.054476] pending: addrconf_verify_work [ 3220.054487] workqueue knbd0-recv: flags=0x1a [ 3220.054490] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3220.054496] in-flight: 655:recv_work [ 3220.054505] workqueue ptlrpc_pinger: flags=0x2 [ 3220.054508] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3220.054513] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3220.055091] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1195s workers=4 idle: 36396 38 11188 [ 3220.055238] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3210s workers=2 idle: 658 [ 3220.370843] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3220.460487] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3220.472373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3220.500387] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3220.526255] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3220.527997] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3220.566416] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3220.583471] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3220.596394] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3220.611191] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3220.617207] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3220.630262] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3220.645833] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3220.660022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3220.676816] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3220.687364] Call Trace: [ 3220.692649] [ 3220.695772] ? show_regs.cold.9+0x22/0x2f [ 3220.713771] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3220.715576] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3220.732461] ? watchdog+0x40/0x40 [ 3220.740443] ? hrtimer_interrupt+0x128/0x2c0 [ 3220.747173] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3220.748895] ? apic_timer_interrupt+0xf/0x20 [ 3220.759047] [ 3220.764564] ? native_safe_halt+0x2e/0x50 [ 3220.778176] kvm_wait+0xcd/0xe0 [ 3220.782643] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3220.797612] do_raw_spin_lock+0x177/0x190 [ 3220.807356] _raw_spin_lock+0x21/0x30 [ 3220.808782] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3220.820930] process_one_work+0x2c8/0x700 [ 3220.826435] worker_thread+0x296/0x6e0 [ 3220.835188] ? rescuer_thread+0x570/0x570 [ 3220.841161] kthread+0x1d1/0x200 [ 3220.847740] ? set_kthread_struct+0x70/0x70 [ 3220.858483] ret_from_fork+0x1f/0x30 [ 3228.055745] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3228.063438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3228.113494] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3228.128355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3228.140640] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3228.151660] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3228.171155] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3228.179372] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3228.187143] RDX: 00000000000020b6 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3228.194265] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3228.199364] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3228.207968] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3228.214961] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3228.233932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3228.236155] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3228.244481] Call Trace: [ 3228.245409] [ 3228.246162] ? show_regs.cold.9+0x22/0x2f [ 3228.259804] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3228.263821] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3228.279990] ? watchdog+0x40/0x40 [ 3228.288715] ? hrtimer_interrupt+0x128/0x2c0 [ 3228.292125] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3228.302857] ? apic_timer_interrupt+0xf/0x20 [ 3228.307139] [ 3228.310864] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3228.331665] do_raw_spin_lock+0x177/0x190 [ 3228.335301] _raw_spin_lock+0x21/0x30 [ 3228.348674] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3228.353053] ? do_raw_spin_unlock+0x75/0x190 [ 3228.356680] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3228.364149] ? do_wait_intr+0xf0/0xf0 [ 3228.366233] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3228.370141] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3228.373764] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3228.377245] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3228.381071] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3228.386039] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3228.393316] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3228.398985] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3228.404130] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3228.412826] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3228.419230] lmv_intent_open+0x338/0x1150 [lmv] [ 3228.422969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3228.427869] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3228.430043] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3228.442871] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3228.447849] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3228.452705] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3228.455893] ? do_raw_spin_unlock+0x75/0x190 [ 3228.458830] ? _raw_spin_unlock+0x12/0x30 [ 3228.463647] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3228.468434] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3228.471836] ll_atomic_open+0x36a/0xd70 [lustre] [ 3228.476924] lookup_open+0x356/0xb40 [ 3228.478267] do_last+0x313/0xfc0 [ 3228.481897] ? nd_jump_root+0xe5/0x160 [ 3228.485835] ? path_init+0x437/0x520 [ 3228.487802] path_openat+0xf7/0x500 [ 3228.491548] ? do_raw_spin_unlock+0x75/0x190 [ 3228.494857] do_filp_open+0x99/0x140 [ 3228.498197] ? getname_flags+0x6e/0x330 [ 3228.500134] ? __check_object_size+0xff/0x256 [ 3228.503868] ? do_raw_spin_unlock+0x75/0x190 [ 3228.508124] ? _raw_spin_unlock+0x12/0x30 [ 3228.510658] do_sys_openat2+0x2b4/0x410 [ 3228.514549] do_sys_open+0x73/0xa0 [ 3228.515827] __x64_sys_openat+0x24/0x30 [ 3228.519355] do_syscall_64+0xc1/0x440 [ 3228.520722] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3228.529873] RIP: 0033:0x7fdc0afbf332 [ 3228.543958] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3228.570751] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3228.588738] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3228.602345] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3228.609350] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3228.617098] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3228.628834] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3240.054705] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3240.066548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3240.096676] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3240.116419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3240.121675] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3240.124603] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3240.141599] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3240.146324] RAX: 0000000000004884 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3240.150437] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3240.154727] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3240.164484] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3240.169979] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3240.175048] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3240.181838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3240.185556] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3240.193808] Call Trace: [ 3240.195857] [ 3240.196693] ? show_regs.cold.9+0x22/0x2f [ 3240.198001] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3240.199308] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3240.202516] ? watchdog+0x40/0x40 [ 3240.205544] ? hrtimer_interrupt+0x128/0x2c0 [ 3240.212056] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3240.213801] ? apic_timer_interrupt+0xf/0x20 [ 3240.220351] [ 3240.220761] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3240.223443] do_raw_spin_lock+0x177/0x190 [ 3240.225292] _raw_spin_lock+0x21/0x30 [ 3240.226453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3240.230003] ? do_raw_spin_unlock+0x75/0x190 [ 3240.235008] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3240.246821] ? do_wait_intr+0xf0/0xf0 [ 3240.252129] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3240.260467] mdc_close+0x53b/0xd70 [mdc] [ 3240.266441] lmv_close+0x18d/0x490 [lmv] [ 3240.268028] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3240.280685] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3240.282416] ll_md_close+0x274/0x800 [lustre] [ 3240.283604] ll_file_release+0x128/0xc50 [lustre] [ 3240.293499] __fput+0x10d/0x440 [ 3240.294250] ____fput+0x12/0x20 [ 3240.295144] task_work_run+0xc6/0x110 [ 3240.304173] exit_to_usermode_loop+0x1dd/0x1f0 [ 3240.307443] do_syscall_64+0x3d6/0x440 [ 3240.314712] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3240.324562] RIP: 0033:0x7fdc0afbfcd8 [ 3240.330047] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3240.345799] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3240.365305] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3240.379567] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3240.402223] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3240.412553] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3240.415374] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3248.048614] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3248.059335] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3248.065205] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3248.065286] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3248.065292] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3248.065295] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3248.065309] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3248.065313] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3248.065317] RAX: 0000000000006f1c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3248.065319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3248.065322] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3248.065324] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3248.065326] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3248.065329] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3248.065332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3248.065334] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3248.065343] Call Trace: [ 3248.065347] [ 3248.065352] ? show_regs.cold.9+0x22/0x2f [ 3248.065360] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3248.065367] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3248.065371] ? watchdog+0x40/0x40 [ 3248.065375] ? hrtimer_interrupt+0x128/0x2c0 [ 3248.065379] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3248.065387] ? apic_timer_interrupt+0xf/0x20 [ 3248.065391] [ 3248.065394] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3248.065398] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3248.065402] ? dump_stack+0xf6/0x10e [ 3248.065407] do_raw_spin_lock+0x177/0x190 [ 3248.065411] _raw_spin_lock+0x21/0x30 [ 3248.065416] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3248.065780] obd_mod_free+0x46/0x140 [ptlrpc] [ 3248.065990] mdc_commit_open+0x91/0xb0 [mdc] [ 3248.066032] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3248.066224] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3248.066411] after_reply+0xe2c/0x1780 [ptlrpc] [ 3248.066598] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3248.066804] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3248.067025] ? do_wait_intr+0xf0/0xf0 [ 3248.067035] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3248.067300] kthread+0x1d1/0x200 [ 3248.067306] ? set_kthread_struct+0x70/0x70 [ 3248.067311] ret_from_fork+0x1f/0x30 [ 3248.411966] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3248.468866] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3248.473631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3248.478622] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3248.483097] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3248.489747] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3248.505153] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3248.508911] RAX: 0000000000005590 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3248.513448] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3248.520853] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3248.525991] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3248.531454] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3248.536800] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3248.539992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3248.543499] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3248.549465] Call Trace: [ 3248.551469] [ 3248.552309] ? show_regs.cold.9+0x22/0x2f [ 3248.553547] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3248.555840] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3248.557959] ? watchdog+0x40/0x40 [ 3248.558929] ? hrtimer_interrupt+0x128/0x2c0 [ 3248.562033] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3248.566837] ? apic_timer_interrupt+0xf/0x20 [ 3248.570646] [ 3248.571899] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3248.576318] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3248.580256] do_raw_spin_lock+0x177/0x190 [ 3248.584425] _raw_spin_lock+0x21/0x30 [ 3248.586907] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3248.590763] process_one_work+0x2c8/0x700 [ 3248.592104] worker_thread+0x296/0x6e0 [ 3248.593717] ? rescuer_thread+0x570/0x570 [ 3248.594956] kthread+0x1d1/0x200 [ 3248.596474] ? set_kthread_struct+0x70/0x70 [ 3248.601346] ret_from_fork+0x1f/0x30 [ 3250.143039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1225s! [ 3250.147130] Showing busy workqueues and worker pools: [ 3250.151981] workqueue events: flags=0x0 [ 3250.155372] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3250.158932] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3250.162232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3250.165425] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3250.171260] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3250.174467] pending: kfree_rcu_monitor [ 3250.179158] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3250.183816] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3250.194203] workqueue events_long: flags=0x0 [ 3250.196832] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3250.201029] pending: lu_objects_flush [obdclass] [ 3250.205209] workqueue events_unbound: flags=0x2 [ 3250.214175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3250.222061] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3250.235835] workqueue events_power_efficient: flags=0x80 [ 3250.241773] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3250.245624] pending: neigh_periodic_work, check_lifetime [ 3250.249980] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3250.254841] pending: do_cache_clean, neigh_periodic_work [ 3250.259059] workqueue mm_percpu_wq: flags=0x8 [ 3250.261277] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3250.267132] pending: lru_add_drain_per_cpu [ 3250.269485] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3250.272847] pending: vmstat_update, lru_add_drain_per_cpu [ 3250.275871] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3250.281119] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3250.284456] workqueue writeback: flags=0x4a [ 3250.286296] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3250.291376] pending: wb_workfn, wb_workfn [ 3250.293272] workqueue xprtiod: flags=0xa [ 3250.295238] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3250.297781] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3250.300498] workqueue ipv6_addrconf: flags=0x40008 [ 3250.302711] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3250.306184] pending: addrconf_verify_work [ 3250.307426] workqueue knbd0-recv: flags=0x1a [ 3250.309385] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3250.312674] in-flight: 655:recv_work [ 3250.314146] workqueue ptlrpc_pinger: flags=0x2 [ 3250.318462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3250.325723] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3250.327519] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1225s workers=4 idle: 36396 38 11188 [ 3250.331680] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3240s workers=2 idle: 658 [ 3256.052013] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3256.056647] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3256.079678] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3256.085012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3256.090301] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3256.092816] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3256.100995] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3256.104137] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3256.107290] RDX: 00000000000067a8 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3256.110747] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3256.113825] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3256.116686] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3256.119852] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3256.123577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3256.126976] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3256.130537] Call Trace: [ 3256.131527] [ 3256.132721] ? show_regs.cold.9+0x22/0x2f [ 3256.134599] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3256.136872] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3256.139643] ? watchdog+0x40/0x40 [ 3256.142237] ? hrtimer_interrupt+0x128/0x2c0 [ 3256.143996] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3256.146343] ? apic_timer_interrupt+0xf/0x20 [ 3256.148282] [ 3256.150082] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3256.152498] do_raw_spin_lock+0x177/0x190 [ 3256.154553] _raw_spin_lock+0x21/0x30 [ 3256.156531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3256.159090] ? do_raw_spin_unlock+0x75/0x190 [ 3256.161350] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3256.164112] ? do_wait_intr+0xf0/0xf0 [ 3256.165526] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3256.168568] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3256.170911] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3256.173544] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3256.175972] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3256.178710] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3256.181104] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3256.183263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3256.186208] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3256.189541] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3256.192898] lmv_intent_open+0x338/0x1150 [lmv] [ 3256.195131] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3256.197994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3256.201967] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3256.204516] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3256.206993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3256.210171] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3256.213037] ? do_raw_spin_unlock+0x75/0x190 [ 3256.214954] ? _raw_spin_unlock+0x12/0x30 [ 3256.216798] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3256.219207] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3256.221454] ll_atomic_open+0x36a/0xd70 [lustre] [ 3256.223842] lookup_open+0x356/0xb40 [ 3256.226768] do_last+0x313/0xfc0 [ 3256.228212] ? nd_jump_root+0xe5/0x160 [ 3256.230073] ? path_init+0x437/0x520 [ 3256.231755] path_openat+0xf7/0x500 [ 3256.233514] ? do_raw_spin_unlock+0x75/0x190 [ 3256.235667] do_filp_open+0x99/0x140 [ 3256.237811] ? getname_flags+0x6e/0x330 [ 3256.240798] ? __check_object_size+0xff/0x256 [ 3256.252701] ? do_raw_spin_unlock+0x75/0x190 [ 3256.254938] ? _raw_spin_unlock+0x12/0x30 [ 3256.258080] do_sys_openat2+0x2b4/0x410 [ 3256.260200] do_sys_open+0x73/0xa0 [ 3256.262681] __x64_sys_openat+0x24/0x30 [ 3256.265117] do_syscall_64+0xc1/0x440 [ 3256.267193] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3256.269933] RIP: 0033:0x7fdc0afbf332 [ 3256.271753] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3256.281910] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3256.285397] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3256.288842] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3256.292256] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3256.296695] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3256.300091] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3268.054053] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3268.057502] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3268.078262] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3268.083504] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3268.087411] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3268.089881] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3268.097955] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3268.101329] RAX: 0000000000003640 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3268.104400] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3268.107485] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3268.110638] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3268.113968] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3268.116962] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3268.120645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3268.123197] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3268.126151] Call Trace: [ 3268.131692] [ 3268.132542] ? show_regs.cold.9+0x22/0x2f [ 3268.134142] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3268.136079] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3268.138370] ? watchdog+0x40/0x40 [ 3268.139782] ? hrtimer_interrupt+0x128/0x2c0 [ 3268.141718] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3268.143981] ? apic_timer_interrupt+0xf/0x20 [ 3268.145974] [ 3268.146869] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3268.149523] do_raw_spin_lock+0x177/0x190 [ 3268.151357] _raw_spin_lock+0x21/0x30 [ 3268.153079] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3268.155564] ? do_raw_spin_unlock+0x75/0x190 [ 3268.157882] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3268.160115] ? do_wait_intr+0xf0/0xf0 [ 3268.162538] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3268.164860] mdc_close+0x53b/0xd70 [mdc] [ 3268.166334] lmv_close+0x18d/0x490 [lmv] [ 3268.168394] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3268.171246] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3268.173821] ll_md_close+0x274/0x800 [lustre] [ 3268.176115] ll_file_release+0x128/0xc50 [lustre] [ 3268.178205] __fput+0x10d/0x440 [ 3268.179952] ____fput+0x12/0x20 [ 3268.181428] task_work_run+0xc6/0x110 [ 3268.183334] exit_to_usermode_loop+0x1dd/0x1f0 [ 3268.185167] do_syscall_64+0x3d6/0x440 [ 3268.187374] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3268.189723] RIP: 0033:0x7fdc0afbfcd8 [ 3268.191435] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3268.200266] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3268.203731] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3268.207045] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3268.212971] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3268.216142] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3268.219613] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3276.048563] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 3276.052324] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 3276.065529] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3276.065609] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3276.065617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3276.065619] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3276.065634] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3276.065639] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3276.065643] RAX: 0000000000001e0a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3276.065645] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3276.065647] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3276.065649] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3276.065652] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3276.065655] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3276.065658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3276.065660] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3276.065670] Call Trace: [ 3276.065675] [ 3276.065679] ? show_regs.cold.9+0x22/0x2f [ 3276.065701] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3276.065709] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3276.065714] ? watchdog+0x40/0x40 [ 3276.065717] ? hrtimer_interrupt+0x128/0x2c0 [ 3276.065722] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3276.065730] ? apic_timer_interrupt+0xf/0x20 [ 3276.065734] [ 3276.065737] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3276.065742] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3276.065745] ? dump_stack+0xf6/0x10e [ 3276.065750] do_raw_spin_lock+0x177/0x190 [ 3276.065755] _raw_spin_lock+0x21/0x30 [ 3276.065760] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3276.066016] obd_mod_free+0x46/0x140 [ptlrpc] [ 3276.066211] mdc_commit_open+0x91/0xb0 [mdc] [ 3276.066248] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3276.066442] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3276.066668] after_reply+0xe2c/0x1780 [ptlrpc] [ 3276.066948] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3276.067141] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3276.067344] ? do_wait_intr+0xf0/0xf0 [ 3276.067353] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3276.067568] kthread+0x1d1/0x200 [ 3276.067574] ? set_kthread_struct+0x70/0x70 [ 3276.067578] ret_from_fork+0x1f/0x30 [ 3276.354284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3276.381567] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3276.388199] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3276.393005] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3276.396337] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3276.399848] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3276.408778] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3276.412220] RAX: 0000000000001206 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3276.415382] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3276.418583] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3276.421805] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3276.425284] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3276.428798] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3276.432575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3276.435205] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3276.438824] Call Trace: [ 3276.443628] [ 3276.447214] ? show_regs.cold.9+0x22/0x2f [ 3276.449049] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3276.454004] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3276.456617] ? watchdog+0x40/0x40 [ 3276.458061] ? hrtimer_interrupt+0x128/0x2c0 [ 3276.460141] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3276.462333] ? apic_timer_interrupt+0xf/0x20 [ 3276.464276] [ 3276.465351] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3276.467790] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3276.470905] do_raw_spin_lock+0x177/0x190 [ 3276.472671] _raw_spin_lock+0x21/0x30 [ 3276.474297] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3276.480814] process_one_work+0x2c8/0x700 [ 3276.484799] worker_thread+0x296/0x6e0 [ 3276.488117] ? rescuer_thread+0x570/0x570 [ 3276.491834] kthread+0x1d1/0x200 [ 3276.495219] ? set_kthread_struct+0x70/0x70 [ 3276.500246] ret_from_fork+0x1f/0x30 [ 3280.351293] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1255s! [ 3280.360622] Showing busy workqueues and worker pools: [ 3280.365993] workqueue events: flags=0x0 [ 3280.372559] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3280.379907] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3280.386744] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3280.391486] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3280.401668] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3280.410398] pending: kfree_rcu_monitor [ 3280.416536] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3280.424551] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3280.433369] workqueue events_long: flags=0x0 [ 3280.435722] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3280.440445] pending: lu_objects_flush [obdclass] [ 3280.447120] workqueue events_unbound: flags=0x2 [ 3280.453250] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3280.460123] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3280.474472] workqueue events_power_efficient: flags=0x80 [ 3280.479967] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3280.489401] pending: neigh_periodic_work, check_lifetime [ 3280.498148] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3280.504965] pending: do_cache_clean, neigh_periodic_work [ 3280.515615] workqueue mm_percpu_wq: flags=0x8 [ 3280.522134] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3280.530429] pending: lru_add_drain_per_cpu [ 3280.532374] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3280.542759] pending: vmstat_update, lru_add_drain_per_cpu [ 3280.545638] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3280.553166] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3280.559442] workqueue writeback: flags=0x4a [ 3280.565442] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3280.569918] pending: wb_workfn, wb_workfn [ 3280.576129] workqueue xprtiod: flags=0xa [ 3280.577638] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3280.588473] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3280.595722] workqueue ipv6_addrconf: flags=0x40008 [ 3280.602759] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3280.611308] pending: addrconf_verify_work [ 3280.615204] workqueue knbd0-recv: flags=0x1a [ 3280.618386] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3280.623167] in-flight: 655:recv_work [ 3280.625968] workqueue ptlrpc_pinger: flags=0x2 [ 3280.629423] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3280.633074] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3280.637794] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1255s workers=4 idle: 36396 38 11188 [ 3280.648717] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3271s workers=2 idle: 658 [ 3284.053770] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3284.056894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3284.096808] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3284.105096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3284.109006] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3284.118299] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3284.132061] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3284.152981] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3284.159961] RDX: 0000000000002180 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3284.164706] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3284.169108] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3284.177248] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3284.184017] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3284.190436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3284.193651] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3284.197137] Call Trace: [ 3284.198490] [ 3284.199583] ? show_regs.cold.9+0x22/0x2f [ 3284.202564] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3284.207621] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3284.215600] ? watchdog+0x40/0x40 [ 3284.218536] ? hrtimer_interrupt+0x128/0x2c0 [ 3284.228284] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3284.232443] ? apic_timer_interrupt+0xf/0x20 [ 3284.234606] [ 3284.235574] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3284.238121] do_raw_spin_lock+0x177/0x190 [ 3284.242875] _raw_spin_lock+0x21/0x30 [ 3284.245126] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3284.252114] ? do_raw_spin_unlock+0x75/0x190 [ 3284.254282] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3284.258721] ? do_wait_intr+0xf0/0xf0 [ 3284.262497] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3284.268515] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3284.277180] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3284.283424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3284.289801] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3284.293131] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3284.299020] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3284.306963] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3284.309801] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3284.316266] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3284.326155] lmv_intent_open+0x338/0x1150 [lmv] [ 3284.328531] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3284.332360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3284.341478] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3284.343974] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3284.347599] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3284.359905] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3284.366620] ? do_raw_spin_unlock+0x75/0x190 [ 3284.371245] ? _raw_spin_unlock+0x12/0x30 [ 3284.373190] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3284.377352] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3284.384243] ll_atomic_open+0x36a/0xd70 [lustre] [ 3284.389605] lookup_open+0x356/0xb40 [ 3284.391542] do_last+0x313/0xfc0 [ 3284.394327] ? nd_jump_root+0xe5/0x160 [ 3284.396080] ? path_init+0x437/0x520 [ 3284.397719] path_openat+0xf7/0x500 [ 3284.400177] ? do_raw_spin_unlock+0x75/0x190 [ 3284.405730] do_filp_open+0x99/0x140 [ 3284.407402] ? getname_flags+0x6e/0x330 [ 3284.411074] ? __check_object_size+0xff/0x256 [ 3284.412870] ? do_raw_spin_unlock+0x75/0x190 [ 3284.414822] ? _raw_spin_unlock+0x12/0x30 [ 3284.417233] do_sys_openat2+0x2b4/0x410 [ 3284.418828] do_sys_open+0x73/0xa0 [ 3284.420430] __x64_sys_openat+0x24/0x30 [ 3284.421867] do_syscall_64+0xc1/0x440 [ 3284.423446] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3284.438027] RIP: 0033:0x7fdc0afbf332 [ 3284.439335] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3284.460236] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3284.469498] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3284.484998] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3284.495352] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3284.509518] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3284.520863] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3296.054074] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3296.056883] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3296.071670] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3296.075003] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3296.077867] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3296.079183] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3296.084882] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3296.086924] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3296.089026] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3296.091322] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3296.094410] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3296.098085] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3296.099945] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3296.102664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3296.104647] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3296.106850] Call Trace: [ 3296.107840] [ 3296.108325] ? show_regs.cold.9+0x22/0x2f [ 3296.109374] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3296.111311] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3296.113285] ? watchdog+0x40/0x40 [ 3296.114871] ? hrtimer_interrupt+0x128/0x2c0 [ 3296.117081] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3296.122113] ? apic_timer_interrupt+0xf/0x20 [ 3296.126970] [ 3296.128642] ? native_safe_halt+0x2e/0x50 [ 3296.130438] kvm_wait+0xcd/0xe0 [ 3296.133953] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3296.136566] do_raw_spin_lock+0x177/0x190 [ 3296.140478] _raw_spin_lock+0x21/0x30 [ 3296.142440] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3296.144910] ? do_raw_spin_unlock+0x75/0x190 [ 3296.147361] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3296.150084] ? do_wait_intr+0xf0/0xf0 [ 3296.151816] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3296.155039] mdc_close+0x53b/0xd70 [mdc] [ 3296.157093] lmv_close+0x18d/0x490 [lmv] [ 3296.158996] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3296.160922] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3296.163206] ll_md_close+0x274/0x800 [lustre] [ 3296.165386] ll_file_release+0x128/0xc50 [lustre] [ 3296.167426] __fput+0x10d/0x440 [ 3296.168307] ____fput+0x12/0x20 [ 3296.169079] task_work_run+0xc6/0x110 [ 3296.170003] exit_to_usermode_loop+0x1dd/0x1f0 [ 3296.173549] do_syscall_64+0x3d6/0x440 [ 3296.175638] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3296.178102] RIP: 0033:0x7fdc0afbfcd8 [ 3296.180262] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3296.187357] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3296.190442] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3296.193002] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3296.195270] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3296.198225] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3296.200818] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3304.051020] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 3304.053990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3304.071046] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3304.078764] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3304.082319] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3304.084829] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3304.087455] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 3304.095820] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3304.098862] RAX: 0000000000004e33 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3304.101747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3304.104779] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3304.107764] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3304.110883] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3304.113503] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3304.118897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3304.121239] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3304.124738] Call Trace: [ 3304.125803] [ 3304.126847] ? show_regs.cold.9+0x22/0x2f [ 3304.128784] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3304.130880] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3304.133226] ? watchdog+0x40/0x40 [ 3304.140460] ? hrtimer_interrupt+0x128/0x2c0 [ 3304.144617] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3304.152848] ? apic_timer_interrupt+0xf/0x20 [ 3304.158990] [ 3304.160411] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3304.163338] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3304.166303] do_raw_spin_lock+0x177/0x190 [ 3304.167454] _raw_spin_lock+0x21/0x30 [ 3304.168569] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3304.173151] process_one_work+0x2c8/0x700 [ 3304.176859] worker_thread+0x296/0x6e0 [ 3304.179008] ? rescuer_thread+0x570/0x570 [ 3304.181224] kthread+0x1d1/0x200 [ 3304.182644] ? set_kthread_struct+0x70/0x70 [ 3304.184854] ret_from_fork+0x1f/0x30 [ 3308.049349] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3308.052189] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3308.098665] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3308.112699] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3308.128483] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3308.137749] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3308.156723] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3308.162709] RAX: 0000000000007ca3 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3308.167873] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3308.174460] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3308.180499] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3308.185592] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3308.189967] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3308.199595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3308.205017] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3308.211861] Call Trace: [ 3308.213707] [ 3308.215756] ? show_regs.cold.9+0x22/0x2f [ 3308.219836] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3308.221618] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3308.238784] ? watchdog+0x40/0x40 [ 3308.244720] ? hrtimer_interrupt+0x128/0x2c0 [ 3308.262615] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3308.270788] ? apic_timer_interrupt+0xf/0x20 [ 3308.277602] [ 3308.281355] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3308.289491] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3308.295665] ? dump_stack+0xf6/0x10e [ 3308.303966] do_raw_spin_lock+0x177/0x190 [ 3308.305132] _raw_spin_lock+0x21/0x30 [ 3308.313732] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3308.319772] obd_mod_free+0x46/0x140 [ptlrpc] [ 3308.329083] mdc_commit_open+0x91/0xb0 [mdc] [ 3308.333151] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3308.337276] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3308.341425] after_reply+0xe2c/0x1780 [ptlrpc] [ 3308.345114] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3308.348402] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3308.351113] ? do_wait_intr+0xf0/0xf0 [ 3308.353075] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3308.357037] kthread+0x1d1/0x200 [ 3308.359631] ? set_kthread_struct+0x70/0x70 [ 3308.361425] ret_from_fork+0x1f/0x30 [ 3311.071557] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1286s! [ 3311.080686] Showing busy workqueues and worker pools: [ 3311.082678] workqueue events: flags=0x0 [ 3311.086104] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3311.090021] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3311.092556] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3311.098408] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3311.105596] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3311.111630] pending: kfree_rcu_monitor [ 3311.115075] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3311.118191] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3311.128090] workqueue events_long: flags=0x0 [ 3311.129307] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3311.136012] pending: lu_objects_flush [obdclass] [ 3311.137954] workqueue events_unbound: flags=0x2 [ 3311.141565] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3311.149118] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3311.156449] workqueue events_power_efficient: flags=0x80 [ 3311.161513] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3311.163359] pending: neigh_periodic_work, check_lifetime [ 3311.166805] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3311.173104] pending: do_cache_clean, neigh_periodic_work [ 3311.175287] workqueue mm_percpu_wq: flags=0x8 [ 3311.177951] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3311.179911] pending: lru_add_drain_per_cpu [ 3311.181543] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3311.188824] pending: vmstat_update, lru_add_drain_per_cpu [ 3311.191253] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3311.194460] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3311.198899] workqueue writeback: flags=0x4a [ 3311.203519] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3311.209069] pending: wb_workfn, wb_workfn [ 3311.212231] workqueue xprtiod: flags=0xa [ 3311.215667] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3311.219280] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3311.224558] workqueue ipv6_addrconf: flags=0x40008 [ 3311.229220] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3311.233355] pending: addrconf_verify_work [ 3311.237717] workqueue knbd0-recv: flags=0x1a [ 3311.242241] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3311.248228] in-flight: 655:recv_work [ 3311.250864] workqueue ptlrpc_pinger: flags=0x2 [ 3311.255272] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3311.260740] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3311.266483] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1286s workers=4 idle: 36396 38 11188 [ 3311.271603] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3301s workers=2 idle: 658 [ 3312.052886] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3312.059766] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3312.084974] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3312.089391] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3312.093110] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3312.097906] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3312.125060] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3312.145350] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3312.155204] RDX: 0000000000006e42 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3312.164316] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3312.174689] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3312.184690] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3312.192954] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3312.208799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3312.214046] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3312.219945] Call Trace: [ 3312.223538] [ 3312.224278] ? show_regs.cold.9+0x22/0x2f [ 3312.226973] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3312.230992] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3312.235562] ? watchdog+0x40/0x40 [ 3312.236666] ? hrtimer_interrupt+0x128/0x2c0 [ 3312.243159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3312.245190] ? apic_timer_interrupt+0xf/0x20 [ 3312.253973] [ 3312.254690] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3312.256229] do_raw_spin_lock+0x177/0x190 [ 3312.260515] _raw_spin_lock+0x21/0x30 [ 3312.261979] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3312.267558] ? do_raw_spin_unlock+0x75/0x190 [ 3312.269921] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3312.280129] ? do_wait_intr+0xf0/0xf0 [ 3312.287774] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3312.294233] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3312.308281] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3312.320703] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3312.322427] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3312.334145] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3312.339331] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3312.343799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3312.348174] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3312.352078] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3312.355251] lmv_intent_open+0x338/0x1150 [lmv] [ 3312.360887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3312.364073] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3312.371716] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3312.373063] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3312.378127] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3312.387318] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3312.397155] ? do_raw_spin_unlock+0x75/0x190 [ 3312.405495] ? _raw_spin_unlock+0x12/0x30 [ 3312.407341] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3312.413020] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3312.415145] ll_atomic_open+0x36a/0xd70 [lustre] [ 3312.419050] lookup_open+0x356/0xb40 [ 3312.420185] do_last+0x313/0xfc0 [ 3312.427219] ? nd_jump_root+0xe5/0x160 [ 3312.430179] ? path_init+0x437/0x520 [ 3312.443058] path_openat+0xf7/0x500 [ 3312.447071] ? do_raw_spin_unlock+0x75/0x190 [ 3312.450261] do_filp_open+0x99/0x140 [ 3312.454999] ? getname_flags+0x6e/0x330 [ 3312.458170] ? __check_object_size+0xff/0x256 [ 3312.463025] ? do_raw_spin_unlock+0x75/0x190 [ 3312.468112] ? _raw_spin_unlock+0x12/0x30 [ 3312.474605] do_sys_openat2+0x2b4/0x410 [ 3312.476957] do_sys_open+0x73/0xa0 [ 3312.479967] __x64_sys_openat+0x24/0x30 [ 3312.484012] do_syscall_64+0xc1/0x440 [ 3312.485300] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3312.488642] RIP: 0033:0x7fdc0afbf332 [ 3312.493533] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3312.511523] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3312.519245] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3312.528304] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3312.535282] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3312.544557] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3312.549584] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3324.054019] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3324.058205] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3324.072980] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3324.080057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3324.087947] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3324.092188] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3324.124793] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3324.133948] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3324.139682] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3324.144877] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3324.149756] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3324.158880] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3324.162884] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3324.173221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3324.182969] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3324.185534] Call Trace: [ 3324.191921] [ 3324.194910] ? show_regs.cold.9+0x22/0x2f [ 3324.199592] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3324.205669] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3324.211111] ? watchdog+0x40/0x40 [ 3324.214843] ? hrtimer_interrupt+0x128/0x2c0 [ 3324.219179] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3324.224007] ? apic_timer_interrupt+0xf/0x20 [ 3324.225821] [ 3324.226671] ? native_safe_halt+0x2e/0x50 [ 3324.227968] kvm_wait+0xcd/0xe0 [ 3324.229111] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3324.231877] do_raw_spin_lock+0x177/0x190 [ 3324.233626] _raw_spin_lock+0x21/0x30 [ 3324.235105] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3324.238008] ? do_raw_spin_unlock+0x75/0x190 [ 3324.239547] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3324.243212] ? do_wait_intr+0xf0/0xf0 [ 3324.245150] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3324.248071] mdc_close+0x53b/0xd70 [mdc] [ 3324.249981] lmv_close+0x18d/0x490 [lmv] [ 3324.251954] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3324.257685] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3324.262832] ll_md_close+0x274/0x800 [lustre] [ 3324.266935] ll_file_release+0x128/0xc50 [lustre] [ 3324.272045] __fput+0x10d/0x440 [ 3324.275286] ____fput+0x12/0x20 [ 3324.276248] task_work_run+0xc6/0x110 [ 3324.277502] exit_to_usermode_loop+0x1dd/0x1f0 [ 3324.284135] do_syscall_64+0x3d6/0x440 [ 3324.286117] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3324.288955] RIP: 0033:0x7fdc0afbfcd8 [ 3324.290182] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3324.307832] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3324.322928] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3324.336981] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3324.344279] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3324.351100] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3324.353901] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3332.051030] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 3332.056964] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3332.082620] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3332.089807] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3332.095866] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3332.099273] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3332.103127] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3332.123865] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3332.132298] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3332.140268] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3332.149030] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3332.156922] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3332.162180] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3332.166599] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3332.177434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3332.184364] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3332.189599] Call Trace: [ 3332.190414] [ 3332.199931] ? show_regs.cold.9+0x22/0x2f [ 3332.205011] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3332.213732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3332.215228] ? watchdog+0x40/0x40 [ 3332.227532] ? hrtimer_interrupt+0x128/0x2c0 [ 3332.229154] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3332.241240] ? apic_timer_interrupt+0xf/0x20 [ 3332.242860] [ 3332.244901] ? native_safe_halt+0x2e/0x50 [ 3332.247474] kvm_wait+0xcd/0xe0 [ 3332.248859] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3332.251279] do_raw_spin_lock+0x177/0x190 [ 3332.252895] _raw_spin_lock+0x21/0x30 [ 3332.254575] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3332.257812] process_one_work+0x2c8/0x700 [ 3332.260022] worker_thread+0x296/0x6e0 [ 3332.269078] ? rescuer_thread+0x570/0x570 [ 3332.270887] kthread+0x1d1/0x200 [ 3332.271990] ? set_kthread_struct+0x70/0x70 [ 3332.276564] ret_from_fork+0x1f/0x30 [ 3336.048962] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3336.055751] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3336.092982] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3336.103121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3336.109468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3336.120243] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3336.135780] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3336.142740] RAX: 0000000000007c1c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3336.148834] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3336.155017] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3336.162573] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3336.166812] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3336.172582] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3336.181373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3336.185954] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3336.195609] Call Trace: [ 3336.199381] [ 3336.200004] ? show_regs.cold.9+0x22/0x2f [ 3336.204388] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3336.207897] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3336.211935] ? watchdog+0x40/0x40 [ 3336.215685] ? hrtimer_interrupt+0x128/0x2c0 [ 3336.220428] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3336.224415] ? apic_timer_interrupt+0xf/0x20 [ 3336.229524] [ 3336.231490] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3336.234558] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3336.238946] ? dump_stack+0xf6/0x10e [ 3336.242710] do_raw_spin_lock+0x177/0x190 [ 3336.245996] _raw_spin_lock+0x21/0x30 [ 3336.249455] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3336.253714] obd_mod_free+0x46/0x140 [ptlrpc] [ 3336.258477] mdc_commit_open+0x91/0xb0 [mdc] [ 3336.261787] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3336.268298] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3336.272882] after_reply+0xe2c/0x1780 [ptlrpc] [ 3336.278281] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3336.289960] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3336.300295] ? do_wait_intr+0xf0/0xf0 [ 3336.308624] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3336.316445] kthread+0x1d1/0x200 [ 3336.318907] ? set_kthread_struct+0x70/0x70 [ 3336.320972] ret_from_fork+0x1f/0x30 [ 3339.369101] rcu: INFO: rcu_sched self-detected stall on CPU [ 3339.372340] rcu: 0-...!: (771004 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 3339.377326] (t=1320026 jiffies g=270761 q=2990) [ 3339.379855] rcu: rcu_sched kthread starved for 1314958 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 3339.384555] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3339.390843] rcu: RCU grace-period kthread stack dump: [ 3339.392970] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3339.397636] Call Trace: [ 3339.398800] __schedule+0x351/0xcb0 [ 3339.402297] schedule+0xc0/0x180 [ 3339.405672] schedule_timeout+0xb4/0x190 [ 3339.407463] ? do_raw_spin_unlock+0x75/0x190 [ 3339.409478] ? __next_timer_interrupt+0x160/0x160 [ 3339.411919] ? prepare_to_swait_event+0xf3/0x140 [ 3339.414194] rcu_gp_kthread+0x8dd/0x15b0 [ 3339.416903] ? force_qs_rnp+0x2c0/0x2c0 [ 3339.418774] kthread+0x1d1/0x200 [ 3339.420277] ? set_kthread_struct+0x70/0x70 [ 3339.424118] ret_from_fork+0x1f/0x30 [ 3339.425874] rcu: Stack dump where RCU GP kthread last ran: [ 3339.428601] Sending NMI from CPU 0 to CPUs 3: [ 3339.430399] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 3339.432962] NMI backtrace for cpu 0 [ 3339.434785] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3339.441445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3339.445751] Call Trace: [ 3339.446987] [ 3339.447879] ? dump_stack+0xbb/0x10e [ 3339.449305] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3339.451331] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3339.453528] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3339.455795] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3339.457886] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3339.460298] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3339.462686] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3339.465030] ? update_process_times+0x6d/0xc0 [ 3339.467171] ? tick_sched_handle.isra.18+0x27/0x90 [ 3339.469688] ? tick_sched_timer+0x53/0xb0 [ 3339.471482] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3339.474148] ? tick_sched_handle.isra.18+0x90/0x90 [ 3339.476434] ? hrtimer_interrupt+0x128/0x2c0 [ 3339.478250] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3339.480338] ? apic_timer_interrupt+0xf/0x20 [ 3339.482099] [ 3339.483254] ? native_safe_halt+0x2e/0x50 [ 3339.485131] ? kvm_wait+0xcd/0xe0 [ 3339.486878] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3339.489201] ? dump_stack+0xf6/0x10e [ 3339.490912] ? do_raw_spin_lock+0x177/0x190 [ 3339.492413] ? _raw_spin_lock+0x21/0x30 [ 3339.494330] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3339.496835] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3339.499276] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3339.501812] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3339.504747] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3339.507620] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3339.510037] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3339.512813] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3339.515399] ? do_wait_intr+0xf0/0xf0 [ 3339.517652] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3339.520838] ? kthread+0x1d1/0x200 [ 3339.522338] ? set_kthread_struct+0x70/0x70 [ 3339.524720] ? ret_from_fork+0x1f/0x30 [ 3340.054497] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3340.061919] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3340.090202] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3340.098403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3340.102556] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3340.104359] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3340.110727] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3340.113644] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3340.116436] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3340.119296] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3340.122285] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3340.125105] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3340.130807] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3340.137905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3340.144678] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3340.150136] Call Trace: [ 3340.151543] [ 3340.152234] ? show_regs.cold.9+0x22/0x2f [ 3340.154223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3340.158252] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3340.160076] ? watchdog+0x40/0x40 [ 3340.161431] ? hrtimer_interrupt+0x128/0x2c0 [ 3340.162940] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3340.168966] ? apic_timer_interrupt+0xf/0x20 [ 3340.172417] [ 3340.173931] ? native_safe_halt+0x2e/0x50 [ 3340.175339] kvm_wait+0xcd/0xe0 [ 3340.177674] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3340.182313] do_raw_spin_lock+0x177/0x190 [ 3340.187167] _raw_spin_lock+0x21/0x30 [ 3340.189574] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3340.192747] ? do_raw_spin_unlock+0x75/0x190 [ 3340.194184] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3340.204619] ? do_wait_intr+0xf0/0xf0 [ 3340.206743] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3340.209816] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3340.211905] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3340.214799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3340.217214] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3340.224636] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3340.229826] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3340.232265] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3340.238753] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3340.249159] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3340.251537] lmv_intent_open+0x338/0x1150 [lmv] [ 3340.255878] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3340.258892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3340.261944] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3340.265853] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3340.268779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3340.273883] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3340.280450] ? do_raw_spin_unlock+0x75/0x190 [ 3340.282736] ? _raw_spin_unlock+0x12/0x30 [ 3340.287201] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3340.294917] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3340.297252] ll_atomic_open+0x36a/0xd70 [lustre] [ 3340.299770] lookup_open+0x356/0xb40 [ 3340.302145] do_last+0x313/0xfc0 [ 3340.304669] ? nd_jump_root+0xe5/0x160 [ 3340.306824] ? path_init+0x437/0x520 [ 3340.308738] path_openat+0xf7/0x500 [ 3340.312244] ? do_raw_spin_unlock+0x75/0x190 [ 3340.315199] do_filp_open+0x99/0x140 [ 3340.316914] ? getname_flags+0x6e/0x330 [ 3340.320066] ? __check_object_size+0xff/0x256 [ 3340.323658] ? do_raw_spin_unlock+0x75/0x190 [ 3340.330256] ? _raw_spin_unlock+0x12/0x30 [ 3340.333377] do_sys_openat2+0x2b4/0x410 [ 3340.337568] do_sys_open+0x73/0xa0 [ 3340.339797] __x64_sys_openat+0x24/0x30 [ 3340.345799] do_syscall_64+0xc1/0x440 [ 3340.348936] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3340.355297] RIP: 0033:0x7fdc0afbf332 [ 3340.356805] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3340.367772] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3340.371189] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3340.374104] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3340.377601] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3340.380869] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3340.384343] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3341.281415] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1316s! [ 3341.293467] Showing busy workqueues and worker pools: [ 3341.303694] workqueue events: flags=0x0 [ 3341.307412] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3341.317259] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3341.334640] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3341.339441] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3341.346159] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3341.350758] pending: kfree_rcu_monitor [ 3341.353053] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3341.357433] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3341.364245] workqueue events_long: flags=0x0 [ 3341.366645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3341.369539] pending: lu_objects_flush [obdclass] [ 3341.371650] workqueue events_unbound: flags=0x2 [ 3341.373605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3341.376509] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3341.382412] workqueue events_power_efficient: flags=0x80 [ 3341.385330] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3341.388898] pending: neigh_periodic_work, check_lifetime [ 3341.391803] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3341.395493] pending: do_cache_clean, neigh_periodic_work [ 3341.399769] workqueue mm_percpu_wq: flags=0x8 [ 3341.402235] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3341.405205] pending: lru_add_drain_per_cpu [ 3341.407506] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3341.410822] pending: vmstat_update, lru_add_drain_per_cpu [ 3341.413820] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3341.417175] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3341.420315] workqueue writeback: flags=0x4a [ 3341.422600] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3341.426261] pending: wb_workfn, wb_workfn [ 3341.429425] workqueue xprtiod: flags=0xa [ 3341.431535] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3341.440858] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3341.454009] workqueue ipv6_addrconf: flags=0x40008 [ 3341.456449] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3341.462424] pending: addrconf_verify_work [ 3341.467928] workqueue knbd0-recv: flags=0x1a [ 3341.469563] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3341.473760] in-flight: 655:recv_work [ 3341.476214] workqueue ptlrpc_pinger: flags=0x2 [ 3341.479780] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3341.483400] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3341.486201] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1316s workers=4 idle: 36396 38 11188 [ 3341.490761] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3332s workers=2 idle: 658 [ 3360.052932] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 3360.055707] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3360.104747] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3360.115335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3360.135735] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3360.143802] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3360.150719] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3360.169720] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3360.176746] RAX: 0000000000007f14 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3360.184422] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3360.191189] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3360.199578] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3360.207345] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3360.215205] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3360.223985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3360.229092] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3360.235578] Call Trace: [ 3360.237195] [ 3360.237845] ? show_regs.cold.9+0x22/0x2f [ 3360.243465] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3360.252418] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3360.256929] ? watchdog+0x40/0x40 [ 3360.258105] ? hrtimer_interrupt+0x128/0x2c0 [ 3360.267468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3360.272062] ? apic_timer_interrupt+0xf/0x20 [ 3360.275361] [ 3360.280904] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3360.284702] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3360.288984] do_raw_spin_lock+0x177/0x190 [ 3360.292918] _raw_spin_lock+0x21/0x30 [ 3360.294491] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3360.298354] process_one_work+0x2c8/0x700 [ 3360.299846] worker_thread+0x296/0x6e0 [ 3360.310184] ? rescuer_thread+0x570/0x570 [ 3360.314521] kthread+0x1d1/0x200 [ 3360.317205] ? set_kthread_struct+0x70/0x70 [ 3360.322536] ret_from_fork+0x1f/0x30 [ 3364.054448] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 3364.057985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3364.082003] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3364.086511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3364.089915] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3364.092341] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3364.103796] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3364.107500] RAX: 0000000000005bf8 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3364.110833] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3364.114368] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3364.117610] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3364.120982] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3364.124789] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3364.128504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3364.131198] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3364.134982] Call Trace: [ 3364.136304] [ 3364.137337] ? show_regs.cold.9+0x22/0x2f [ 3364.139375] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3364.141147] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3364.143221] ? watchdog+0x40/0x40 [ 3364.144698] ? hrtimer_interrupt+0x128/0x2c0 [ 3364.147143] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3364.149353] ? apic_timer_interrupt+0xf/0x20 [ 3364.151299] [ 3364.152190] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3364.154764] do_raw_spin_lock+0x177/0x190 [ 3364.156670] _raw_spin_lock+0x21/0x30 [ 3364.158573] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3364.161092] ? do_raw_spin_unlock+0x75/0x190 [ 3364.162983] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3364.165466] ? do_wait_intr+0xf0/0xf0 [ 3364.167070] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3364.169722] mdc_close+0x53b/0xd70 [mdc] [ 3364.171553] lmv_close+0x18d/0x490 [lmv] [ 3364.173049] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3364.175744] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3364.177937] ll_md_close+0x274/0x800 [lustre] [ 3364.180450] ll_file_release+0x128/0xc50 [lustre] [ 3364.183056] __fput+0x10d/0x440 [ 3364.184505] ____fput+0x12/0x20 [ 3364.186016] task_work_run+0xc6/0x110 [ 3364.187973] exit_to_usermode_loop+0x1dd/0x1f0 [ 3364.190413] do_syscall_64+0x3d6/0x440 [ 3364.191979] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3364.193970] RIP: 0033:0x7fdc0afbfcd8 [ 3364.195551] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3364.204723] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3364.208545] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3364.212983] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3364.215349] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3364.218897] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3364.222014] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3368.048491] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3368.052210] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3368.068302] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3368.068778] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3368.068785] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3368.068787] RIP: 0010:load_balance+0x725/0x1090 [ 3368.068799] Code: 48 83 00 01 9c 58 0f 1f 44 00 00 48 83 05 7a 6d dd 02 01 48 83 05 82 b9 dd 02 01 f6 c4 02 0f 85 22 03 00 00 48 8b 3c 24 57 9d <0f> 1f 44 00 00 8b 84 24 98 00 00 00 a8 02 74 23 83 e0 fd 48 83 05 [ 3368.068803] RSP: 0018:ffffba8e40003de0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3368.068807] RAX: 0000000000000006 RBX: ffff9fe615482870 RCX: ffff9fe642033440 [ 3368.068809] RDX: 0000000000000008 RSI: ffff9fe6154827c0 RDI: 0000000000000202 [ 3368.068811] RBP: ffffba8e40003e90 R08: ffff9fe628060000 R09: 0000000000000000 [ 3368.068813] R10: ffff9fe615482858 R11: 0000000000009f31 R12: ffff9fe6154827c0 [ 3368.068815] R13: ffff9fe6420b3e08 R14: 0000000000000001 R15: ffff9fe6420b3380 [ 3368.068818] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3368.068820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3368.068822] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3368.068831] Call Trace: [ 3368.068837] [ 3368.068843] ? show_regs.cold.9+0x22/0x2f [ 3368.068854] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3368.068860] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3368.068864] ? watchdog+0x40/0x40 [ 3368.068868] ? hrtimer_interrupt+0x128/0x2c0 [ 3368.068881] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3368.068890] ? apic_timer_interrupt+0xf/0x20 [ 3368.068896] ? apic_timer_interrupt+0xa/0x20 [ 3368.068901] ? load_balance+0x725/0x1090 [ 3368.068907] rebalance_domains+0x348/0x580 [ 3368.068911] ? kvm_clock_get_cycles+0x2c/0x50 [ 3368.068920] run_rebalance_domains+0x68/0xa0 [ 3368.068924] __do_softirq+0x134/0x4e8 [ 3368.068929] irq_exit_rcu+0x1c6/0x1e0 [ 3368.068935] irq_exit+0x12/0x30 [ 3368.068938] smp_apic_timer_interrupt+0xe0/0x270 [ 3368.068944] apic_timer_interrupt+0xf/0x20 [ 3368.068950] [ 3368.068951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3368.068958] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3368.068961] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3368.068965] RAX: 0000000000007515 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3368.068968] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3368.068969] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3368.068971] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3368.068974] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3368.068979] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3368.068984] ? dump_stack+0xf6/0x10e [ 3368.068989] do_raw_spin_lock+0x177/0x190 [ 3368.068994] _raw_spin_lock+0x21/0x30 [ 3368.069000] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3368.077012] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [ 3368.141905] obd_mod_free+0x46/0x140 [ptlrpc] [ 3368.153256] ec(O) [ 3368.159633] mdc_commit_open+0x91/0xb0 [mdc] [ 3368.159678] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3368.160297] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3368.160495] after_reply+0xe2c/0x1780 [ptlrpc] [ 3368.162131] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3368.162363] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3368.162563] ? do_wait_intr+0xf0/0xf0 [ 3368.162576] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3368.162782] kthread+0x1d1/0x200 [ 3368.162789] ? set_kthread_struct+0x70/0x70 [ 3368.162793] ret_from_fork+0x1f/0x30 [ 3368.484589] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3368.498211] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3368.502948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3368.507283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3368.510292] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3368.519131] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3368.526318] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3368.532510] RDX: 0000000000007c18 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3368.536772] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3368.545783] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3368.554750] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3368.561323] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3368.565964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3368.569524] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3368.575302] Call Trace: [ 3368.580493] [ 3368.581389] ? show_regs.cold.9+0x22/0x2f [ 3368.583565] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3368.586336] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3368.593204] ? watchdog+0x40/0x40 [ 3368.597843] ? hrtimer_interrupt+0x128/0x2c0 [ 3368.604070] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3368.608412] ? apic_timer_interrupt+0xf/0x20 [ 3368.611399] [ 3368.612415] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3368.618282] do_raw_spin_lock+0x177/0x190 [ 3368.624320] _raw_spin_lock+0x21/0x30 [ 3368.630354] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3368.636758] ? do_raw_spin_unlock+0x75/0x190 [ 3368.640031] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3368.644941] ? do_wait_intr+0xf0/0xf0 [ 3368.648408] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3368.652600] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3368.655808] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3368.659137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3368.661117] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3368.664681] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3368.666401] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3368.669472] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3368.671108] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3368.681366] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3368.686661] lmv_intent_open+0x338/0x1150 [lmv] [ 3368.688742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3368.691569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3368.693748] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3368.699721] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3368.701511] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3368.707371] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3368.710900] ? do_raw_spin_unlock+0x75/0x190 [ 3368.715334] ? _raw_spin_unlock+0x12/0x30 [ 3368.720430] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3368.728619] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3368.732831] ll_atomic_open+0x36a/0xd70 [lustre] [ 3368.737013] lookup_open+0x356/0xb40 [ 3368.740856] do_last+0x313/0xfc0 [ 3368.741985] ? nd_jump_root+0xe5/0x160 [ 3368.745420] ? path_init+0x437/0x520 [ 3368.747807] path_openat+0xf7/0x500 [ 3368.751425] ? do_raw_spin_unlock+0x75/0x190 [ 3368.753814] do_filp_open+0x99/0x140 [ 3368.756540] ? getname_flags+0x6e/0x330 [ 3368.759406] ? __check_object_size+0xff/0x256 [ 3368.762189] ? do_raw_spin_unlock+0x75/0x190 [ 3368.764963] ? _raw_spin_unlock+0x12/0x30 [ 3368.767422] do_sys_openat2+0x2b4/0x410 [ 3368.772044] do_sys_open+0x73/0xa0 [ 3368.776668] __x64_sys_openat+0x24/0x30 [ 3368.780747] do_syscall_64+0xc1/0x440 [ 3368.782692] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3368.785119] RIP: 0033:0x7fdc0afbf332 [ 3368.788427] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3368.803484] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3368.809249] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3368.816302] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3368.824380] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3368.837504] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3368.847981] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3371.999514] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1347s! [ 3372.012084] Showing busy workqueues and worker pools: [ 3372.014014] workqueue events: flags=0x0 [ 3372.028433] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3372.038537] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3372.048857] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3372.058078] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3372.071921] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3372.080545] pending: kfree_rcu_monitor [ 3372.084769] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3372.091012] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3372.103648] workqueue events_long: flags=0x0 [ 3372.112312] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3372.120165] pending: lu_objects_flush [obdclass] [ 3372.126155] workqueue events_unbound: flags=0x2 [ 3372.133283] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3372.144829] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3372.163667] workqueue events_power_efficient: flags=0x80 [ 3372.168764] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3372.180278] pending: neigh_periodic_work, check_lifetime [ 3372.184265] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3372.194051] pending: do_cache_clean, neigh_periodic_work [ 3372.201197] workqueue mm_percpu_wq: flags=0x8 [ 3372.207388] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3372.219282] pending: lru_add_drain_per_cpu [ 3372.223010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3372.227210] pending: vmstat_update, lru_add_drain_per_cpu [ 3372.238171] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3372.246798] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3372.248634] workqueue writeback: flags=0x4a [ 3372.257981] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3372.266781] pending: wb_workfn, wb_workfn [ 3372.274127] workqueue xprtiod: flags=0xa [ 3372.282237] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3372.292623] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3372.312871] workqueue ipv6_addrconf: flags=0x40008 [ 3372.314612] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3372.337317] pending: addrconf_verify_work [ 3372.342833] workqueue knbd0-recv: flags=0x1a [ 3372.348018] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3372.351743] in-flight: 655:recv_work [ 3372.360610] workqueue ptlrpc_pinger: flags=0x2 [ 3372.366309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3372.372069] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3372.380864] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1347s workers=4 idle: 36396 38 11188 [ 3372.392591] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3362s workers=2 idle: 658 [ 3388.051217] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3388.063197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3388.157346] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3388.168687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3388.173395] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3388.178708] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3388.185211] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3388.210196] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3388.220704] RAX: 000000000000361f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3388.236364] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3388.243700] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3388.253461] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3388.261549] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3388.266038] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3388.276597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3388.284264] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3388.297397] Call Trace: [ 3388.299405] [ 3388.303849] ? show_regs.cold.9+0x22/0x2f [ 3388.310398] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3388.318688] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3388.322556] ? watchdog+0x40/0x40 [ 3388.333696] ? hrtimer_interrupt+0x128/0x2c0 [ 3388.342816] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3388.350326] ? apic_timer_interrupt+0xf/0x20 [ 3388.357405] [ 3388.359315] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3388.363771] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3388.372697] do_raw_spin_lock+0x177/0x190 [ 3388.380693] _raw_spin_lock+0x21/0x30 [ 3388.388999] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3388.399418] process_one_work+0x2c8/0x700 [ 3388.407316] worker_thread+0x296/0x6e0 [ 3388.411843] ? rescuer_thread+0x570/0x570 [ 3388.418415] kthread+0x1d1/0x200 [ 3388.422616] ? set_kthread_struct+0x70/0x70 [ 3388.432344] ret_from_fork+0x1f/0x30 [ 3392.059754] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 3392.071709] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3392.140596] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3392.157586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3392.167820] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3392.179907] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3392.200272] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3392.207717] RAX: 000000000000558f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3392.214711] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3392.222287] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3392.228642] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3392.237239] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3392.243999] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3392.250318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3392.253677] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3392.257209] Call Trace: [ 3392.258395] [ 3392.259253] ? show_regs.cold.9+0x22/0x2f [ 3392.261194] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3392.263120] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3392.265374] ? watchdog+0x40/0x40 [ 3392.266635] ? hrtimer_interrupt+0x128/0x2c0 [ 3392.268716] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3392.270502] ? apic_timer_interrupt+0xf/0x20 [ 3392.271969] [ 3392.274138] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3392.276529] do_raw_spin_lock+0x177/0x190 [ 3392.278839] _raw_spin_lock+0x21/0x30 [ 3392.280958] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3392.284124] ? do_raw_spin_unlock+0x75/0x190 [ 3392.285816] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3392.287882] ? do_wait_intr+0xf0/0xf0 [ 3392.294782] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3392.296997] mdc_close+0x53b/0xd70 [mdc] [ 3392.298731] lmv_close+0x18d/0x490 [lmv] [ 3392.300253] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3392.302393] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3392.304470] ll_md_close+0x274/0x800 [lustre] [ 3392.306245] ll_file_release+0x128/0xc50 [lustre] [ 3392.308005] __fput+0x10d/0x440 [ 3392.309600] ____fput+0x12/0x20 [ 3392.310719] task_work_run+0xc6/0x110 [ 3392.312177] exit_to_usermode_loop+0x1dd/0x1f0 [ 3392.313812] do_syscall_64+0x3d6/0x440 [ 3392.317429] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3392.321517] RIP: 0033:0x7fdc0afbfcd8 [ 3392.322805] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3392.331714] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3392.334691] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3392.337670] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3392.340899] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3392.344608] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3392.347691] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3396.052160] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3396.055561] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3396.075397] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3396.079449] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3396.082811] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3396.085155] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3396.091659] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3396.094352] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3396.096875] RDX: 00000000000023bc RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3396.099649] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3396.102413] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3396.105025] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3396.107672] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3396.110586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3396.112648] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3396.114967] Call Trace: [ 3396.115863] [ 3396.116627] ? show_regs.cold.9+0x22/0x2f [ 3396.118274] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3396.119816] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3396.121666] ? watchdog+0x40/0x40 [ 3396.122721] ? hrtimer_interrupt+0x128/0x2c0 [ 3396.124205] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3396.125750] ? apic_timer_interrupt+0xf/0x20 [ 3396.127313] [ 3396.127961] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3396.129720] do_raw_spin_lock+0x177/0x190 [ 3396.131363] _raw_spin_lock+0x21/0x30 [ 3396.132530] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3396.135379] ? do_raw_spin_unlock+0x75/0x190 [ 3396.136804] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3396.139191] ? do_wait_intr+0xf0/0xf0 [ 3396.140369] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3396.142422] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3396.144773] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3396.147005] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3396.149428] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3396.151301] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3396.152866] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3396.154401] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3396.156659] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3396.158941] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3396.161268] lmv_intent_open+0x338/0x1150 [lmv] [ 3396.163640] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3396.165679] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3396.167898] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3396.169458] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3396.171657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3396.173520] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3396.175497] ? do_raw_spin_unlock+0x75/0x190 [ 3396.176859] ? _raw_spin_unlock+0x12/0x30 [ 3396.178344] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3396.180187] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3396.182302] ll_atomic_open+0x36a/0xd70 [lustre] [ 3396.184187] lookup_open+0x356/0xb40 [ 3396.185611] do_last+0x313/0xfc0 [ 3396.186714] ? nd_jump_root+0xe5/0x160 [ 3396.187983] ? path_init+0x437/0x520 [ 3396.189359] path_openat+0xf7/0x500 [ 3396.190691] ? do_raw_spin_unlock+0x75/0x190 [ 3396.192489] do_filp_open+0x99/0x140 [ 3396.193847] ? getname_flags+0x6e/0x330 [ 3396.195696] ? __check_object_size+0xff/0x256 [ 3396.197574] ? do_raw_spin_unlock+0x75/0x190 [ 3396.198993] ? _raw_spin_unlock+0x12/0x30 [ 3396.200750] do_sys_openat2+0x2b4/0x410 [ 3396.202223] do_sys_open+0x73/0xa0 [ 3396.203521] __x64_sys_openat+0x24/0x30 [ 3396.204824] do_syscall_64+0xc1/0x440 [ 3396.206216] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3396.207816] RIP: 0033:0x7fdc0afbf332 [ 3396.209074] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3396.216237] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3396.218520] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3396.221170] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3396.223693] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3396.226275] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3396.228829] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3400.051360] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3400.060807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3400.129980] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3400.148596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3400.160618] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3400.162551] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3400.203625] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3400.209784] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3400.217833] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3400.224095] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3400.232768] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3400.239714] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3400.257214] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3400.269713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3400.275182] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3400.282526] Call Trace: [ 3400.285794] [ 3400.286799] ? show_regs.cold.9+0x22/0x2f [ 3400.288809] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3400.297616] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3400.303142] ? watchdog+0x40/0x40 [ 3400.310193] ? hrtimer_interrupt+0x128/0x2c0 [ 3400.314885] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3400.323375] ? apic_timer_interrupt+0xf/0x20 [ 3400.330710] [ 3400.333363] ? native_safe_halt+0x2e/0x50 [ 3400.337883] kvm_wait+0xcd/0xe0 [ 3400.339841] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3400.344789] ? dump_stack+0xf6/0x10e [ 3400.348121] do_raw_spin_lock+0x177/0x190 [ 3400.356364] _raw_spin_lock+0x21/0x30 [ 3400.362448] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3400.371647] obd_mod_free+0x46/0x140 [ptlrpc] [ 3400.387210] mdc_commit_open+0x91/0xb0 [mdc] [ 3400.398437] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3400.404746] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3400.406971] after_reply+0xe2c/0x1780 [ptlrpc] [ 3400.423355] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3400.439645] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3400.450860] ? do_wait_intr+0xf0/0xf0 [ 3400.459989] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3400.471621] kthread+0x1d1/0x200 [ 3400.482961] ? set_kthread_struct+0x70/0x70 [ 3400.484543] ret_from_fork+0x1f/0x30 [ 3402.723489] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1377s! [ 3402.731123] Showing busy workqueues and worker pools: [ 3402.737013] workqueue events: flags=0x0 [ 3402.744887] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3402.753609] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3402.758795] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3402.765807] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3402.780292] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3402.793871] pending: kfree_rcu_monitor [ 3402.796298] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3402.807357] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3402.817503] workqueue events_long: flags=0x0 [ 3402.819077] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3402.828968] pending: lu_objects_flush [obdclass] [ 3402.838073] workqueue events_unbound: flags=0x2 [ 3402.840767] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3402.852504] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3402.865950] workqueue events_power_efficient: flags=0x80 [ 3402.878569] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3402.886681] pending: neigh_periodic_work, check_lifetime [ 3402.895952] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3402.901025] pending: do_cache_clean, neigh_periodic_work [ 3402.905233] workqueue mm_percpu_wq: flags=0x8 [ 3402.912946] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3402.917077] pending: lru_add_drain_per_cpu [ 3402.919456] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3402.922055] pending: vmstat_update, lru_add_drain_per_cpu [ 3402.924779] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3402.932426] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3402.940675] workqueue writeback: flags=0x4a [ 3402.943049] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3402.951787] pending: wb_workfn, wb_workfn [ 3402.958187] workqueue xprtiod: flags=0xa [ 3402.960026] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3402.964417] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3402.968467] workqueue ipv6_addrconf: flags=0x40008 [ 3402.970840] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3402.973695] pending: addrconf_verify_work [ 3402.975843] workqueue knbd0-recv: flags=0x1a [ 3402.978579] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3402.981075] in-flight: 655:recv_work [ 3402.983559] workqueue ptlrpc_pinger: flags=0x2 [ 3402.985669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3402.989005] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3403.001318] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1378s workers=4 idle: 36396 38 11188 [ 3403.013766] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3393s workers=2 idle: 658 [ 3416.051619] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3416.056321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3416.093478] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3416.099968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3416.103179] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3416.109811] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3416.112348] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3416.119862] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3416.122732] RAX: 0000000000002a22 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3416.125480] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3416.128625] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3416.131901] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3416.135798] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3416.139282] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3416.142734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3416.145794] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3416.148425] Call Trace: [ 3416.149522] [ 3416.150317] ? show_regs.cold.9+0x22/0x2f [ 3416.151936] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3416.153663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3416.155813] ? watchdog+0x40/0x40 [ 3416.157040] ? hrtimer_interrupt+0x128/0x2c0 [ 3416.159115] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3416.161190] ? apic_timer_interrupt+0xf/0x20 [ 3416.163122] [ 3416.164182] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3416.166200] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3416.168511] do_raw_spin_lock+0x177/0x190 [ 3416.174801] _raw_spin_lock+0x21/0x30 [ 3416.176784] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3416.179562] process_one_work+0x2c8/0x700 [ 3416.181498] worker_thread+0x296/0x6e0 [ 3416.183143] ? rescuer_thread+0x570/0x570 [ 3416.185964] kthread+0x1d1/0x200 [ 3416.187313] ? set_kthread_struct+0x70/0x70 [ 3416.189161] ret_from_fork+0x1f/0x30 [ 3420.054008] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 3420.059130] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3420.087339] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3420.101316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3420.115360] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3420.128273] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3420.153924] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3420.161786] RAX: 00000000000055be RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3420.171158] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3420.176272] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3420.183498] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3420.187667] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3420.198215] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3420.209320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3420.211295] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3420.219416] Call Trace: [ 3420.220892] [ 3420.222047] ? show_regs.cold.9+0x22/0x2f [ 3420.224882] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3420.227794] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3420.231713] ? watchdog+0x40/0x40 [ 3420.233492] ? hrtimer_interrupt+0x128/0x2c0 [ 3420.242297] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3420.248326] ? apic_timer_interrupt+0xf/0x20 [ 3420.256401] [ 3420.258300] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3420.262532] do_raw_spin_lock+0x177/0x190 [ 3420.275356] _raw_spin_lock+0x21/0x30 [ 3420.280023] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3420.295026] ? do_raw_spin_unlock+0x75/0x190 [ 3420.297707] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3420.301936] ? do_wait_intr+0xf0/0xf0 [ 3420.304114] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3420.306725] mdc_close+0x53b/0xd70 [mdc] [ 3420.308409] lmv_close+0x18d/0x490 [lmv] [ 3420.319179] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3420.326008] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3420.332700] ll_md_close+0x274/0x800 [lustre] [ 3420.335903] ll_file_release+0x128/0xc50 [lustre] [ 3420.340380] __fput+0x10d/0x440 [ 3420.342880] ____fput+0x12/0x20 [ 3420.345939] task_work_run+0xc6/0x110 [ 3420.348823] exit_to_usermode_loop+0x1dd/0x1f0 [ 3420.351966] do_syscall_64+0x3d6/0x440 [ 3420.355911] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3420.361739] RIP: 0033:0x7fdc0afbfcd8 [ 3420.366198] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3420.404816] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3420.419065] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3420.425976] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3420.445791] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3420.462478] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3420.473810] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3424.061189] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3424.066194] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3424.092250] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3424.098233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3424.102651] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3424.105928] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3424.118916] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3424.138518] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3424.148389] RDX: 000000000000615b RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3424.154137] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3424.159269] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3424.165832] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3424.169616] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3424.179981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3424.189962] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3424.205862] Call Trace: [ 3424.206879] [ 3424.207780] ? show_regs.cold.9+0x22/0x2f [ 3424.209781] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3424.211843] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3424.213723] ? watchdog+0x40/0x40 [ 3424.215275] ? hrtimer_interrupt+0x128/0x2c0 [ 3424.216914] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3424.221575] ? apic_timer_interrupt+0xf/0x20 [ 3424.225419] [ 3424.226277] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3424.231029] do_raw_spin_lock+0x177/0x190 [ 3424.236384] _raw_spin_lock+0x21/0x30 [ 3424.237303] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3424.238535] ? do_raw_spin_unlock+0x75/0x190 [ 3424.241226] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3424.244830] ? do_wait_intr+0xf0/0xf0 [ 3424.247267] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3424.249382] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3424.252435] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3424.254758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3424.257216] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3424.258759] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3424.260430] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3424.262564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3424.265499] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3424.268568] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3424.270132] lmv_intent_open+0x338/0x1150 [lmv] [ 3424.272111] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3424.274195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3424.276297] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3424.278022] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3424.280439] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3424.282343] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3424.283998] ? do_raw_spin_unlock+0x75/0x190 [ 3424.289552] ? _raw_spin_unlock+0x12/0x30 [ 3424.290488] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3424.291642] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3424.292756] ll_atomic_open+0x36a/0xd70 [lustre] [ 3424.295792] lookup_open+0x356/0xb40 [ 3424.297021] do_last+0x313/0xfc0 [ 3424.298588] ? nd_jump_root+0xe5/0x160 [ 3424.301128] ? path_init+0x437/0x520 [ 3424.302950] path_openat+0xf7/0x500 [ 3424.305176] ? do_raw_spin_unlock+0x75/0x190 [ 3424.306955] do_filp_open+0x99/0x140 [ 3424.308956] ? getname_flags+0x6e/0x330 [ 3424.310875] ? __check_object_size+0xff/0x256 [ 3424.312986] ? do_raw_spin_unlock+0x75/0x190 [ 3424.315196] ? _raw_spin_unlock+0x12/0x30 [ 3424.317033] do_sys_openat2+0x2b4/0x410 [ 3424.318793] do_sys_open+0x73/0xa0 [ 3424.320207] __x64_sys_openat+0x24/0x30 [ 3424.323032] do_syscall_64+0xc1/0x440 [ 3424.325100] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3424.328020] RIP: 0033:0x7fdc0afbf332 [ 3424.329288] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3424.340452] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3424.345335] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3424.349147] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3424.353241] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3424.357503] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3424.361442] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3428.048925] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3428.056018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3428.090734] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3428.095660] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3428.099412] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3428.101970] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3428.113780] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3428.118237] RAX: 00000000000059fc RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3428.120551] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3428.128042] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3428.131551] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3428.140646] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3428.144687] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3428.151774] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3428.154377] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3428.159979] Call Trace: [ 3428.161615] [ 3428.163230] ? show_regs.cold.9+0x22/0x2f [ 3428.164664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3428.167996] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3428.169626] ? watchdog+0x40/0x40 [ 3428.171024] ? hrtimer_interrupt+0x128/0x2c0 [ 3428.175372] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3428.176660] ? apic_timer_interrupt+0xf/0x20 [ 3428.178160] [ 3428.182570] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3428.187097] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3428.193093] ? dump_stack+0xf6/0x10e [ 3428.196642] do_raw_spin_lock+0x177/0x190 [ 3428.200491] _raw_spin_lock+0x21/0x30 [ 3428.203023] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3428.205858] obd_mod_free+0x46/0x140 [ptlrpc] [ 3428.211096] mdc_commit_open+0x91/0xb0 [mdc] [ 3428.215055] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3428.221353] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3428.226768] after_reply+0xe2c/0x1780 [ptlrpc] [ 3428.234045] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3428.238356] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3428.240650] ? do_wait_intr+0xf0/0xf0 [ 3428.242609] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3428.246889] kthread+0x1d1/0x200 [ 3428.250535] ? set_kthread_struct+0x70/0x70 [ 3428.254699] ret_from_fork+0x1f/0x30 [ 3433.439044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1408s! [ 3433.446762] Showing busy workqueues and worker pools: [ 3433.452223] workqueue events: flags=0x0 [ 3433.455013] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3433.465526] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3433.468124] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3433.481858] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3433.488593] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3433.504046] pending: kfree_rcu_monitor [ 3433.510981] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3433.529722] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3433.550603] workqueue events_long: flags=0x0 [ 3433.560203] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3433.572092] pending: lu_objects_flush [obdclass] [ 3433.584569] workqueue events_unbound: flags=0x2 [ 3433.592490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3433.595948] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3433.617307] workqueue events_power_efficient: flags=0x80 [ 3433.629090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3433.638962] pending: neigh_periodic_work, check_lifetime [ 3433.647169] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3433.656937] pending: do_cache_clean, neigh_periodic_work [ 3433.661030] workqueue mm_percpu_wq: flags=0x8 [ 3433.663175] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3433.674168] pending: lru_add_drain_per_cpu [ 3433.680577] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3433.686860] pending: vmstat_update, lru_add_drain_per_cpu [ 3433.692132] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3433.699294] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3433.706431] workqueue writeback: flags=0x4a [ 3433.710887] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3433.715432] pending: wb_workfn, wb_workfn [ 3433.719753] workqueue xprtiod: flags=0xa [ 3433.726380] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3433.733885] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3433.744019] workqueue ipv6_addrconf: flags=0x40008 [ 3433.747453] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3433.752828] pending: addrconf_verify_work [ 3433.756174] workqueue knbd0-recv: flags=0x1a [ 3433.759890] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3433.764909] in-flight: 655:recv_work [ 3433.768080] workqueue ptlrpc_pinger: flags=0x2 [ 3433.770689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3433.775732] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3433.780142] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1408s workers=4 idle: 36396 38 11188 [ 3433.784156] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3424s workers=2 idle: 658 [ 3444.051352] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3444.064456] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3444.115228] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3444.119727] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3444.123245] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3444.125638] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3444.128107] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3444.135640] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3444.142274] RAX: 0000000000007514 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3444.146681] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3444.150730] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3444.156505] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3444.161740] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3444.165569] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3444.174514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3444.177193] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3444.185739] Call Trace: [ 3444.189299] [ 3444.190697] ? show_regs.cold.9+0x22/0x2f [ 3444.195931] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3444.200583] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3444.204721] ? watchdog+0x40/0x40 [ 3444.206610] ? hrtimer_interrupt+0x128/0x2c0 [ 3444.209769] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3444.214471] ? apic_timer_interrupt+0xf/0x20 [ 3444.215903] [ 3444.216488] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3444.222989] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3444.227775] do_raw_spin_lock+0x177/0x190 [ 3444.236641] _raw_spin_lock+0x21/0x30 [ 3444.242711] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3444.251355] process_one_work+0x2c8/0x700 [ 3444.257844] worker_thread+0x296/0x6e0 [ 3444.265111] ? rescuer_thread+0x570/0x570 [ 3444.267611] kthread+0x1d1/0x200 [ 3444.270217] ? set_kthread_struct+0x70/0x70 [ 3444.272240] ret_from_fork+0x1f/0x30 [ 3448.054024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 3448.059764] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3448.093707] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3448.106002] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3448.116418] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3448.122450] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3448.135896] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3448.144692] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3448.148009] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3448.155970] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3448.159547] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3448.162718] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3448.165781] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3448.170955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3448.176205] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3448.179291] Call Trace: [ 3448.180348] [ 3448.181609] ? show_regs.cold.9+0x22/0x2f [ 3448.182930] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3448.184895] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3448.186634] ? watchdog+0x40/0x40 [ 3448.187840] ? hrtimer_interrupt+0x128/0x2c0 [ 3448.189261] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3448.190868] ? apic_timer_interrupt+0xf/0x20 [ 3448.192402] [ 3448.193174] ? native_safe_halt+0x2e/0x50 [ 3448.194959] kvm_wait+0xcd/0xe0 [ 3448.197762] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3448.200157] do_raw_spin_lock+0x177/0x190 [ 3448.201800] _raw_spin_lock+0x21/0x30 [ 3448.203410] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3448.205734] ? do_raw_spin_unlock+0x75/0x190 [ 3448.208146] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3448.210384] ? do_wait_intr+0xf0/0xf0 [ 3448.211701] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3448.213701] mdc_close+0x53b/0xd70 [mdc] [ 3448.215319] lmv_close+0x18d/0x490 [lmv] [ 3448.216859] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3448.219437] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3448.221468] ll_md_close+0x274/0x800 [lustre] [ 3448.223370] ll_file_release+0x128/0xc50 [lustre] [ 3448.225225] __fput+0x10d/0x440 [ 3448.226635] ____fput+0x12/0x20 [ 3448.227977] task_work_run+0xc6/0x110 [ 3448.229642] exit_to_usermode_loop+0x1dd/0x1f0 [ 3448.231640] do_syscall_64+0x3d6/0x440 [ 3448.233193] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3448.238082] RIP: 0033:0x7fdc0afbfcd8 [ 3448.240455] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3448.278768] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3448.291665] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3448.302250] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3448.305926] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3448.308878] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3448.311407] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3452.052518] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3452.064717] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3452.087104] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3452.091626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3452.097340] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3452.102597] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3452.111285] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3452.116309] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3452.119677] RDX: 0000000000007abc RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3452.124303] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3452.128493] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3452.132395] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3452.136066] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3452.139912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3452.144565] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3452.148025] Call Trace: [ 3452.148973] [ 3452.149674] ? show_regs.cold.9+0x22/0x2f [ 3452.151623] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3452.153753] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3452.155504] ? watchdog+0x40/0x40 [ 3452.158579] ? hrtimer_interrupt+0x128/0x2c0 [ 3452.160269] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3452.164277] ? apic_timer_interrupt+0xf/0x20 [ 3452.166692] [ 3452.167692] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3452.170180] do_raw_spin_lock+0x177/0x190 [ 3452.172214] _raw_spin_lock+0x21/0x30 [ 3452.173843] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3452.177138] ? do_raw_spin_unlock+0x75/0x190 [ 3452.178693] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3452.182475] ? do_wait_intr+0xf0/0xf0 [ 3452.184269] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3452.187393] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3452.192257] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3452.196687] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3452.200182] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3452.203243] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3452.205128] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3452.207045] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3452.209503] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3452.212335] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3452.215925] lmv_intent_open+0x338/0x1150 [lmv] [ 3452.217821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3452.220465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3452.223613] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3452.225530] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3452.227666] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3452.230098] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3452.232757] ? do_raw_spin_unlock+0x75/0x190 [ 3452.234407] ? _raw_spin_unlock+0x12/0x30 [ 3452.235981] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3452.237868] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3452.240093] ll_atomic_open+0x36a/0xd70 [lustre] [ 3452.241678] lookup_open+0x356/0xb40 [ 3452.243284] do_last+0x313/0xfc0 [ 3452.244687] ? nd_jump_root+0xe5/0x160 [ 3452.246129] ? path_init+0x437/0x520 [ 3452.247654] path_openat+0xf7/0x500 [ 3452.249034] ? do_raw_spin_unlock+0x75/0x190 [ 3452.250782] do_filp_open+0x99/0x140 [ 3452.252164] ? getname_flags+0x6e/0x330 [ 3452.253473] ? __check_object_size+0xff/0x256 [ 3452.255274] ? do_raw_spin_unlock+0x75/0x190 [ 3452.257172] ? _raw_spin_unlock+0x12/0x30 [ 3452.258686] do_sys_openat2+0x2b4/0x410 [ 3452.260328] do_sys_open+0x73/0xa0 [ 3452.261903] __x64_sys_openat+0x24/0x30 [ 3452.264062] do_syscall_64+0xc1/0x440 [ 3452.266392] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3452.268595] RIP: 0033:0x7fdc0afbf332 [ 3452.270351] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3452.277639] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3452.280708] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3452.283318] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3452.285990] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3452.288694] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3452.291575] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3460.048069] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3460.057084] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3460.095747] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3460.105769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3460.115475] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3460.118289] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3460.134599] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3460.137146] RAX: 0000000000003b0b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3460.149391] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3460.152797] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3460.158421] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3460.163211] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3460.173344] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3460.177674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3460.181329] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3460.185817] Call Trace: [ 3460.188404] [ 3460.189575] ? show_regs.cold.9+0x22/0x2f [ 3460.194634] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3460.197724] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3460.199875] ? watchdog+0x40/0x40 [ 3460.204601] ? hrtimer_interrupt+0x128/0x2c0 [ 3460.210846] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3460.214270] ? apic_timer_interrupt+0xf/0x20 [ 3460.217037] [ 3460.218944] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3460.221939] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3460.225767] ? dump_stack+0xf6/0x10e [ 3460.227711] do_raw_spin_lock+0x177/0x190 [ 3460.229958] _raw_spin_lock+0x21/0x30 [ 3460.232947] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3460.241305] obd_mod_free+0x46/0x140 [ptlrpc] [ 3460.248604] mdc_commit_open+0x91/0xb0 [mdc] [ 3460.255714] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3460.264842] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3460.272843] after_reply+0xe2c/0x1780 [ptlrpc] [ 3460.281866] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3460.285704] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3460.292485] ? do_wait_intr+0xf0/0xf0 [ 3460.293800] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3460.300967] kthread+0x1d1/0x200 [ 3460.303590] ? set_kthread_struct+0x70/0x70 [ 3460.306816] ret_from_fork+0x1f/0x30 [ 3464.159097] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1439s! [ 3464.162094] Showing busy workqueues and worker pools: [ 3464.166048] workqueue events: flags=0x0 [ 3464.170130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3464.176530] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3464.182218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3464.189200] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3464.206222] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3464.217739] pending: kfree_rcu_monitor [ 3464.227404] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3464.229667] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3464.253616] workqueue events_long: flags=0x0 [ 3464.257718] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3464.268539] pending: lu_objects_flush [obdclass] [ 3464.277668] workqueue events_unbound: flags=0x2 [ 3464.281630] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3464.284707] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3464.291144] workqueue events_power_efficient: flags=0x80 [ 3464.295183] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3464.300540] pending: neigh_periodic_work, check_lifetime [ 3464.307289] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3464.310531] pending: do_cache_clean, neigh_periodic_work [ 3464.313751] workqueue mm_percpu_wq: flags=0x8 [ 3464.315551] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3464.318547] pending: lru_add_drain_per_cpu [ 3464.320721] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3464.324120] pending: vmstat_update, lru_add_drain_per_cpu [ 3464.328230] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3464.332684] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3464.337662] workqueue writeback: flags=0x4a [ 3464.339525] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3464.342322] pending: wb_workfn, wb_workfn [ 3464.344237] workqueue xprtiod: flags=0xa [ 3464.346853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3464.353862] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3464.360669] workqueue ipv6_addrconf: flags=0x40008 [ 3464.363053] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3464.366624] pending: addrconf_verify_work [ 3464.370110] workqueue knbd0-recv: flags=0x1a [ 3464.371522] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3464.375768] in-flight: 655:recv_work [ 3464.379053] workqueue ptlrpc_pinger: flags=0x2 [ 3464.380912] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3464.384477] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3464.388444] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1439s workers=4 idle: 36396 38 11188 [ 3464.392672] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3454s workers=2 idle: 658 [ 3472.051195] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3472.070154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3472.140992] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3472.157260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3472.164106] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3472.170070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3472.175750] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3472.193748] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3472.199235] RAX: 0000000000003155 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3472.209724] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3472.216515] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3472.229021] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3472.233482] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3472.239899] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3472.244970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3472.251279] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3472.258854] Call Trace: [ 3472.260232] [ 3472.261200] ? show_regs.cold.9+0x22/0x2f [ 3472.265631] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3472.267370] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3472.268978] ? watchdog+0x40/0x40 [ 3472.270974] ? hrtimer_interrupt+0x128/0x2c0 [ 3472.272617] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3472.277230] ? apic_timer_interrupt+0xf/0x20 [ 3472.280182] [ 3472.283716] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3472.285411] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3472.294985] do_raw_spin_lock+0x177/0x190 [ 3472.296929] _raw_spin_lock+0x21/0x30 [ 3472.301497] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3472.303967] process_one_work+0x2c8/0x700 [ 3472.306101] worker_thread+0x296/0x6e0 [ 3472.307852] ? rescuer_thread+0x570/0x570 [ 3472.310432] kthread+0x1d1/0x200 [ 3472.313424] ? set_kthread_struct+0x70/0x70 [ 3472.315716] ret_from_fork+0x1f/0x30 [ 3476.054685] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 3476.060667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3476.097085] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3476.107420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3476.116516] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3476.121956] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3476.139497] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3476.148106] RAX: 0000000000003b94 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3476.154689] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3476.159782] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3476.166657] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3476.173485] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3476.179854] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3476.186891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3476.191489] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3476.198916] Call Trace: [ 3476.201272] [ 3476.203132] ? show_regs.cold.9+0x22/0x2f [ 3476.205575] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3476.211430] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3476.221999] ? watchdog+0x40/0x40 [ 3476.223229] ? hrtimer_interrupt+0x128/0x2c0 [ 3476.228145] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3476.238011] ? apic_timer_interrupt+0xf/0x20 [ 3476.245504] [ 3476.252500] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3476.254304] do_raw_spin_lock+0x177/0x190 [ 3476.268677] _raw_spin_lock+0x21/0x30 [ 3476.274015] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3476.280341] ? do_raw_spin_unlock+0x75/0x190 [ 3476.287026] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3476.292471] ? do_wait_intr+0xf0/0xf0 [ 3476.296400] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3476.303217] mdc_close+0x53b/0xd70 [mdc] [ 3476.307399] lmv_close+0x18d/0x490 [lmv] [ 3476.311133] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3476.318791] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3476.328653] ll_md_close+0x274/0x800 [lustre] [ 3476.335746] ll_file_release+0x128/0xc50 [lustre] [ 3476.340676] __fput+0x10d/0x440 [ 3476.344534] ____fput+0x12/0x20 [ 3476.348666] task_work_run+0xc6/0x110 [ 3476.352149] exit_to_usermode_loop+0x1dd/0x1f0 [ 3476.354366] do_syscall_64+0x3d6/0x440 [ 3476.359073] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3476.363348] RIP: 0033:0x7fdc0afbfcd8 [ 3476.367571] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3476.387730] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3476.395496] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3476.403054] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3476.409722] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3476.414339] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3476.425898] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3480.052396] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 3480.055289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3480.073192] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3480.077343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3480.085708] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3480.087879] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3480.099382] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3480.103171] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3480.106854] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3480.116049] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3480.121041] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3480.127062] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3480.131443] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3480.137097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3480.141993] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3480.146933] Call Trace: [ 3480.148012] [ 3480.149879] ? show_regs.cold.9+0x22/0x2f [ 3480.156936] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3480.160962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3480.170389] ? watchdog+0x40/0x40 [ 3480.171650] ? hrtimer_interrupt+0x128/0x2c0 [ 3480.177034] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3480.182891] ? apic_timer_interrupt+0xf/0x20 [ 3480.189624] [ 3480.191711] ? native_safe_halt+0x2e/0x50 [ 3480.195893] kvm_wait+0xcd/0xe0 [ 3480.197004] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3480.201012] do_raw_spin_lock+0x177/0x190 [ 3480.203967] _raw_spin_lock+0x21/0x30 [ 3480.205864] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3480.211974] ? do_raw_spin_unlock+0x75/0x190 [ 3480.213659] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3480.216688] ? do_wait_intr+0xf0/0xf0 [ 3480.218248] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3480.220723] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3480.223246] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3480.225971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3480.229040] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3480.231463] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3480.237190] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3480.238843] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3480.243415] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3480.250590] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3480.252550] lmv_intent_open+0x338/0x1150 [lmv] [ 3480.262696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3480.268050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3480.283067] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3480.287893] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3480.291114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3480.297980] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3480.301271] ? do_raw_spin_unlock+0x75/0x190 [ 3480.304464] ? _raw_spin_unlock+0x12/0x30 [ 3480.307480] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3480.311705] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3480.314552] ll_atomic_open+0x36a/0xd70 [lustre] [ 3480.317482] lookup_open+0x356/0xb40 [ 3480.321497] do_last+0x313/0xfc0 [ 3480.325204] ? nd_jump_root+0xe5/0x160 [ 3480.330664] ? path_init+0x437/0x520 [ 3480.332883] path_openat+0xf7/0x500 [ 3480.335090] ? do_raw_spin_unlock+0x75/0x190 [ 3480.337164] do_filp_open+0x99/0x140 [ 3480.339197] ? getname_flags+0x6e/0x330 [ 3480.340979] ? __check_object_size+0xff/0x256 [ 3480.343561] ? do_raw_spin_unlock+0x75/0x190 [ 3480.345306] ? _raw_spin_unlock+0x12/0x30 [ 3480.347614] do_sys_openat2+0x2b4/0x410 [ 3480.350048] do_sys_open+0x73/0xa0 [ 3480.351578] __x64_sys_openat+0x24/0x30 [ 3480.353320] do_syscall_64+0xc1/0x440 [ 3480.355240] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3480.361211] RIP: 0033:0x7fdc0afbf332 [ 3480.365630] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3480.374852] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3480.379183] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3480.382899] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3480.385993] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3480.388987] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3480.395698] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3492.048886] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 3492.061390] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3492.084011] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3492.088961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3492.094670] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3492.099008] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3492.111846] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3492.117093] RAX: 000000000000730c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3492.120337] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3492.125888] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3492.134204] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3492.143183] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3492.154302] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3492.169232] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3492.176048] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3492.181821] Call Trace: [ 3492.184274] [ 3492.186766] ? show_regs.cold.9+0x22/0x2f [ 3492.193019] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3492.195193] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3492.198161] ? watchdog+0x40/0x40 [ 3492.199397] ? hrtimer_interrupt+0x128/0x2c0 [ 3492.203425] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3492.204740] ? apic_timer_interrupt+0xf/0x20 [ 3492.205800] [ 3492.206398] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3492.209984] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3492.221578] ? dump_stack+0xf6/0x10e [ 3492.225031] do_raw_spin_lock+0x177/0x190 [ 3492.228671] _raw_spin_lock+0x21/0x30 [ 3492.231779] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3492.235848] obd_mod_free+0x46/0x140 [ptlrpc] [ 3492.239674] mdc_commit_open+0x91/0xb0 [mdc] [ 3492.242577] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3492.248420] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3492.252981] after_reply+0xe2c/0x1780 [ptlrpc] [ 3492.261706] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3492.265789] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3492.267387] ? do_wait_intr+0xf0/0xf0 [ 3492.269985] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3492.274695] kthread+0x1d1/0x200 [ 3492.279665] ? set_kthread_struct+0x70/0x70 [ 3492.282843] ret_from_fork+0x1f/0x30 [ 3494.879630] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1469s! [ 3494.887918] Showing busy workqueues and worker pools: [ 3494.892881] workqueue events: flags=0x0 [ 3494.901619] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3494.916260] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3494.930488] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3494.939162] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3494.949959] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3494.953639] pending: kfree_rcu_monitor [ 3494.958151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3494.965603] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3494.975411] workqueue events_long: flags=0x0 [ 3494.980763] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3494.990418] pending: lu_objects_flush [obdclass] [ 3495.000393] workqueue events_unbound: flags=0x2 [ 3495.005695] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3495.012028] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3495.018208] workqueue events_power_efficient: flags=0x80 [ 3495.021302] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3495.025744] pending: neigh_periodic_work, check_lifetime [ 3495.030552] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3495.035951] pending: do_cache_clean, neigh_periodic_work [ 3495.038943] workqueue mm_percpu_wq: flags=0x8 [ 3495.044594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3495.051059] pending: lru_add_drain_per_cpu [ 3495.053566] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3495.058676] pending: vmstat_update, lru_add_drain_per_cpu [ 3495.060996] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3495.063314] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3495.066684] workqueue writeback: flags=0x4a [ 3495.071331] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3495.075597] pending: wb_workfn, wb_workfn [ 3495.078227] workqueue xprtiod: flags=0xa [ 3495.080194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3495.083460] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3495.087188] workqueue ipv6_addrconf: flags=0x40008 [ 3495.089281] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3495.092513] pending: addrconf_verify_work [ 3495.094658] workqueue knbd0-recv: flags=0x1a [ 3495.099727] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3495.109484] in-flight: 655:recv_work [ 3495.112835] workqueue ptlrpc_pinger: flags=0x2 [ 3495.117783] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3495.125166] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3495.133459] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1470s workers=4 idle: 36396 38 11188 [ 3495.146047] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3485s workers=2 idle: 658 [ 3500.051025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3500.055009] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3500.072648] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3500.076584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3500.080021] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3500.082346] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3500.083955] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3500.090390] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3500.093024] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3500.095578] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3500.098110] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3500.100284] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3500.102665] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3500.105284] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3500.107871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3500.109908] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3500.112320] Call Trace: [ 3500.113116] [ 3500.114006] ? show_regs.cold.9+0x22/0x2f [ 3500.115302] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3500.117227] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3500.118650] ? watchdog+0x40/0x40 [ 3500.120003] ? hrtimer_interrupt+0x128/0x2c0 [ 3500.121463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3500.123450] ? apic_timer_interrupt+0xf/0x20 [ 3500.124759] [ 3500.125444] ? native_safe_halt+0x2e/0x50 [ 3500.127006] kvm_wait+0xcd/0xe0 [ 3500.128140] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3500.129936] do_raw_spin_lock+0x177/0x190 [ 3500.131317] _raw_spin_lock+0x21/0x30 [ 3500.132632] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3500.135300] process_one_work+0x2c8/0x700 [ 3500.136688] worker_thread+0x296/0x6e0 [ 3500.138156] ? rescuer_thread+0x570/0x570 [ 3500.139653] kthread+0x1d1/0x200 [ 3500.141175] ? set_kthread_struct+0x70/0x70 [ 3500.142421] ret_from_fork+0x1f/0x30 [ 3504.054029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3504.056874] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3504.074980] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3504.079165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3504.082730] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3504.084711] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3504.092727] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3504.095749] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3504.098729] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3504.102369] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3504.105581] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3504.108291] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3504.111071] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3504.114173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3504.116438] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3504.119561] Call Trace: [ 3504.120581] [ 3504.121324] ? show_regs.cold.9+0x22/0x2f [ 3504.123070] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3504.125985] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3504.127917] ? watchdog+0x40/0x40 [ 3504.129175] ? hrtimer_interrupt+0x128/0x2c0 [ 3504.130839] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3504.132702] ? apic_timer_interrupt+0xf/0x20 [ 3504.134303] [ 3504.135052] ? native_safe_halt+0x2e/0x50 [ 3504.136674] kvm_wait+0xcd/0xe0 [ 3504.137752] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3504.139382] do_raw_spin_lock+0x177/0x190 [ 3504.140825] _raw_spin_lock+0x21/0x30 [ 3504.142410] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3504.144635] ? do_raw_spin_unlock+0x75/0x190 [ 3504.146200] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3504.148684] ? do_wait_intr+0xf0/0xf0 [ 3504.150416] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3504.152253] mdc_close+0x53b/0xd70 [mdc] [ 3504.154005] lmv_close+0x18d/0x490 [lmv] [ 3504.155699] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3504.157833] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3504.159774] ll_md_close+0x274/0x800 [lustre] [ 3504.161616] ll_file_release+0x128/0xc50 [lustre] [ 3504.163653] __fput+0x10d/0x440 [ 3504.164788] ____fput+0x12/0x20 [ 3504.165876] task_work_run+0xc6/0x110 [ 3504.167254] exit_to_usermode_loop+0x1dd/0x1f0 [ 3504.168874] do_syscall_64+0x3d6/0x440 [ 3504.170657] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3504.172405] RIP: 0033:0x7fdc0afbfcd8 [ 3504.173779] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3504.181206] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3504.183971] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3504.186834] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3504.189286] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3504.192281] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3504.194900] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3508.052642] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 3508.061905] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3508.104894] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3508.111084] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3508.132713] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3508.144846] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3508.168810] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3508.179289] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3508.192670] RDX: 0000000000003336 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3508.201199] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3508.209637] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3508.213325] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3508.217557] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3508.224710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3508.230912] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3508.241858] Call Trace: [ 3508.244371] [ 3508.251197] ? show_regs.cold.9+0x22/0x2f [ 3508.252470] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3508.254145] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3508.274770] ? watchdog+0x40/0x40 [ 3508.275985] ? hrtimer_interrupt+0x128/0x2c0 [ 3508.285970] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3508.287732] ? apic_timer_interrupt+0xf/0x20 [ 3508.297341] [ 3508.298082] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3508.303277] do_raw_spin_lock+0x177/0x190 [ 3508.308719] _raw_spin_lock+0x21/0x30 [ 3508.315483] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3508.320160] ? do_raw_spin_unlock+0x75/0x190 [ 3508.324851] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3508.330565] ? do_wait_intr+0xf0/0xf0 [ 3508.331819] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3508.340472] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3508.343945] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3508.348324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3508.354898] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3508.361949] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3508.363523] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3508.368123] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3508.371317] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3508.383270] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3508.389248] lmv_intent_open+0x338/0x1150 [lmv] [ 3508.395598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3508.398718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3508.404679] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3508.409591] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3508.411830] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3508.426305] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3508.428935] ? do_raw_spin_unlock+0x75/0x190 [ 3508.433622] ? _raw_spin_unlock+0x12/0x30 [ 3508.438935] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3508.451374] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3508.460744] ll_atomic_open+0x36a/0xd70 [lustre] [ 3508.463689] lookup_open+0x356/0xb40 [ 3508.465760] do_last+0x313/0xfc0 [ 3508.467710] ? nd_jump_root+0xe5/0x160 [ 3508.469484] ? path_init+0x437/0x520 [ 3508.477025] path_openat+0xf7/0x500 [ 3508.482689] ? do_raw_spin_unlock+0x75/0x190 [ 3508.488621] do_filp_open+0x99/0x140 [ 3508.500029] ? getname_flags+0x6e/0x330 [ 3508.505949] ? __check_object_size+0xff/0x256 [ 3508.513932] ? do_raw_spin_unlock+0x75/0x190 [ 3508.515446] ? _raw_spin_unlock+0x12/0x30 [ 3508.521614] do_sys_openat2+0x2b4/0x410 [ 3508.523883] do_sys_open+0x73/0xa0 [ 3508.532709] __x64_sys_openat+0x24/0x30 [ 3508.534061] do_syscall_64+0xc1/0x440 [ 3508.535383] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3508.548365] RIP: 0033:0x7fdc0afbf332 [ 3508.552665] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3508.576935] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3508.583405] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3508.593559] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3508.601648] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3508.614883] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3508.619939] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3519.373785] rcu: INFO: rcu_sched self-detected stall on CPU [ 3519.376692] rcu: 0-...!: (865338 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 3519.382169] (t=1500030 jiffies g=270761 q=3032) [ 3519.384569] rcu: rcu_sched kthread starved for 1494962 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3519.389380] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3519.394262] rcu: RCU grace-period kthread stack dump: [ 3519.396642] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3519.402581] Call Trace: [ 3519.403786] __schedule+0x351/0xcb0 [ 3519.405777] schedule+0xc0/0x180 [ 3519.407468] schedule_timeout+0xb4/0x190 [ 3519.409175] ? do_raw_spin_unlock+0x75/0x190 [ 3519.411138] ? __next_timer_interrupt+0x160/0x160 [ 3519.412652] ? prepare_to_swait_event+0xf3/0x140 [ 3519.435982] rcu_gp_kthread+0x8dd/0x15b0 [ 3519.446751] ? force_qs_rnp+0x2c0/0x2c0 [ 3519.449623] kthread+0x1d1/0x200 [ 3519.451572] ? set_kthread_struct+0x70/0x70 [ 3519.459570] ret_from_fork+0x1f/0x30 [ 3519.461415] rcu: Stack dump where RCU GP kthread last ran: [ 3519.464785] Sending NMI from CPU 0 to CPUs 2: [ 3519.470954] NMI backtrace for cpu 2 [ 3519.471041] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3519.471044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3519.471045] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3519.471046] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3519.471126] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 [ 3519.471130] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3519.471132] RDX: 0000000000004dac RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3519.471133] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3519.471134] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3519.471136] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3519.471137] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3519.471139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3519.471140] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3519.471141] Call Trace: [ 3519.471142] [ 3519.471143] ? show_regs.cold.9+0x22/0x2f [ 3519.471144] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3519.471145] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3519.471146] ? nmi_handle+0xbd/0x210 [ 3519.471147] ? default_do_nmi+0x4b/0x1b0 [ 3519.471148] ? do_nmi+0x280/0x380 [ 3519.471149] ? end_repeat_nmi+0x16/0x1f [ 3519.471151] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3519.471152] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3519.471153] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3519.471154] [ 3519.471155] do_raw_spin_lock+0x177/0x190 [ 3519.471156] _raw_spin_lock+0x21/0x30 [ 3519.471157] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3519.471158] ? do_raw_spin_unlock+0x75/0x190 [ 3519.471160] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3519.471161] ? do_wait_intr+0xf0/0xf0 [ 3519.471162] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3519.471163] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3519.471165] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3519.471166] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3519.471167] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3519.471168] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3519.471169] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3519.471170] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3519.471171] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3519.471173] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3519.471173] lmv_intent_open+0x338/0x1150 [lmv] [ 3519.471174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3519.471176] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3519.471177] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3519.471178] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3519.471179] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3519.471180] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3519.471182] ? do_raw_spin_unlock+0x75/0x190 [ 3519.471182] ? _raw_spin_unlock+0x12/0x30 [ 3519.471184] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3519.471185] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3519.471186] ll_atomic_open+0x36a/0xd70 [lustre] [ 3519.471187] lookup_open+0x356/0xb40 [ 3519.471188] do_last+0x313/0xfc0 [ 3519.471189] ? nd_jump_root+0xe5/0x160 [ 3519.471190] ? path_init+0x437/0x520 [ 3519.471191] path_openat+0xf7/0x500 [ 3519.471192] ? do_raw_spin_unlock+0x75/0x190 [ 3519.471193] do_filp_open+0x99/0x140 [ 3519.471194] ? getname_flags+0x6e/0x330 [ 3519.471195] ? __check_object_size+0xff/0x256 [ 3519.471196] ? do_raw_spin_unlock+0x75/0x190 [ 3519.471197] ? _raw_spin_unlock+0x12/0x30 [ 3519.471198] do_sys_openat2+0x2b4/0x410 [ 3519.471199] do_sys_open+0x73/0xa0 [ 3519.471200] __x64_sys_openat+0x24/0x30 [ 3519.471201] do_syscall_64+0xc1/0x440 [ 3519.471202] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3519.471210] RIP: 0033:0x7fdc0afbf332 [ 3519.471212] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3519.471214] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3519.471217] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3519.471219] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3519.471220] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3519.471231] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3519.471233] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3519.474010] NMI backtrace for cpu 0 [ 3519.862617] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3519.865272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3519.867727] Call Trace: [ 3519.868675] [ 3519.869597] ? dump_stack+0xbb/0x10e [ 3519.870938] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3519.873008] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3519.874781] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3519.876756] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3519.878930] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3519.881341] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3519.883081] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3519.885052] ? update_process_times+0x6d/0xc0 [ 3519.886668] ? tick_sched_handle.isra.18+0x27/0x90 [ 3519.888441] ? tick_sched_timer+0x53/0xb0 [ 3519.889786] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3519.891399] ? tick_sched_handle.isra.18+0x90/0x90 [ 3519.893192] ? hrtimer_interrupt+0x128/0x2c0 [ 3519.894719] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3519.896279] ? apic_timer_interrupt+0xf/0x20 [ 3519.898106] [ 3519.898802] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3519.900685] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3519.902707] ? dump_stack+0xf6/0x10e [ 3519.904094] ? do_raw_spin_lock+0x177/0x190 [ 3519.905438] ? _raw_spin_lock+0x21/0x30 [ 3519.906735] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3519.908936] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3519.910973] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3519.912875] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3519.915121] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3519.917545] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3519.919545] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3519.921621] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3519.923530] ? do_wait_intr+0xf0/0xf0 [ 3519.924764] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3519.926652] ? kthread+0x1d1/0x200 [ 3519.927795] ? set_kthread_struct+0x70/0x70 [ 3519.929368] ? ret_from_fork+0x1f/0x30 [ 3525.599409] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1500s! [ 3525.603285] Showing busy workqueues and worker pools: [ 3525.605984] workqueue events: flags=0x0 [ 3525.609028] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3525.617236] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3525.620933] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3525.630008] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3525.640503] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3525.653954] pending: kfree_rcu_monitor [ 3525.657219] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3525.662906] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3525.671052] workqueue events_long: flags=0x0 [ 3525.675669] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3525.680066] pending: lu_objects_flush [obdclass] [ 3525.684358] workqueue events_unbound: flags=0x2 [ 3525.695832] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3525.701481] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3525.714010] workqueue events_power_efficient: flags=0x80 [ 3525.722813] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3525.733163] pending: neigh_periodic_work, check_lifetime [ 3525.738141] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3525.743543] pending: do_cache_clean, neigh_periodic_work [ 3525.754429] workqueue mm_percpu_wq: flags=0x8 [ 3525.761648] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3525.768736] pending: lru_add_drain_per_cpu [ 3525.781495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3525.791925] pending: vmstat_update, lru_add_drain_per_cpu [ 3525.804491] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3525.815067] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3525.823328] workqueue writeback: flags=0x4a [ 3525.831635] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3525.840333] pending: wb_workfn, wb_workfn [ 3525.848069] workqueue xprtiod: flags=0xa [ 3525.850673] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3525.863509] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3525.874631] workqueue ipv6_addrconf: flags=0x40008 [ 3525.883075] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3525.894395] pending: addrconf_verify_work [ 3525.902301] workqueue knbd0-recv: flags=0x1a [ 3525.903634] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3525.918912] in-flight: 655:recv_work [ 3525.923534] workqueue ptlrpc_pinger: flags=0x2 [ 3525.930013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3525.946185] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3525.967140] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1501s workers=4 idle: 36396 38 11188 [ 3525.980813] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3516s workers=2 idle: 658 [ 3528.059442] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3528.065269] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3528.125013] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3528.139338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3528.154040] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3528.158730] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3528.168547] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3528.197112] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3528.204835] RAX: 000000000000488a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3528.218097] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3528.226254] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3528.228182] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3528.235148] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3528.238934] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3528.242834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3528.245551] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3528.249383] Call Trace: [ 3528.251168] [ 3528.254158] ? show_regs.cold.9+0x22/0x2f [ 3528.261532] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3528.263292] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3528.269979] ? watchdog+0x40/0x40 [ 3528.272011] ? hrtimer_interrupt+0x128/0x2c0 [ 3528.278094] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3528.284001] ? apic_timer_interrupt+0xf/0x20 [ 3528.288348] [ 3528.292896] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3528.300292] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3528.307169] do_raw_spin_lock+0x177/0x190 [ 3528.314825] _raw_spin_lock+0x21/0x30 [ 3528.318137] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3528.327800] process_one_work+0x2c8/0x700 [ 3528.333341] worker_thread+0x296/0x6e0 [ 3528.335645] ? rescuer_thread+0x570/0x570 [ 3528.340711] kthread+0x1d1/0x200 [ 3528.342932] ? set_kthread_struct+0x70/0x70 [ 3528.347004] ret_from_fork+0x1f/0x30 [ 3532.054077] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3532.059666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3532.147276] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3532.174550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3532.186430] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3532.191664] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3532.208199] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3532.219533] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3532.229094] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3532.256173] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3532.269378] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3532.277908] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3532.291620] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3532.295948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3532.310011] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3532.317715] Call Trace: [ 3532.318581] [ 3532.324596] ? show_regs.cold.9+0x22/0x2f [ 3532.328348] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3532.333358] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3532.335945] ? watchdog+0x40/0x40 [ 3532.338343] ? hrtimer_interrupt+0x128/0x2c0 [ 3532.341561] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3532.346371] ? apic_timer_interrupt+0xf/0x20 [ 3532.349988] [ 3532.352254] ? native_safe_halt+0x2e/0x50 [ 3532.354368] kvm_wait+0xcd/0xe0 [ 3532.355987] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3532.358551] do_raw_spin_lock+0x177/0x190 [ 3532.361020] _raw_spin_lock+0x21/0x30 [ 3532.363865] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3532.367053] ? do_raw_spin_unlock+0x75/0x190 [ 3532.368945] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3532.372808] ? do_wait_intr+0xf0/0xf0 [ 3532.377960] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3532.383232] mdc_close+0x53b/0xd70 [mdc] [ 3532.385474] lmv_close+0x18d/0x490 [lmv] [ 3532.387010] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3532.389901] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3532.392024] ll_md_close+0x274/0x800 [lustre] [ 3532.394142] ll_file_release+0x128/0xc50 [lustre] [ 3532.396710] __fput+0x10d/0x440 [ 3532.403831] ____fput+0x12/0x20 [ 3532.405583] task_work_run+0xc6/0x110 [ 3532.407386] exit_to_usermode_loop+0x1dd/0x1f0 [ 3532.409545] do_syscall_64+0x3d6/0x440 [ 3532.411912] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3532.419448] RIP: 0033:0x7fdc0afbfcd8 [ 3532.422849] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3532.436696] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3532.441495] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3532.446877] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3532.454812] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3532.463826] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3532.474010] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3544.052506] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3544.056975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3544.075065] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3544.079519] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3544.086573] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3544.089704] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3544.098870] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3544.101886] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3544.105057] RDX: 00000000000072c8 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3544.107991] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3544.110840] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3544.113567] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3544.116498] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3544.119732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3544.121942] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3544.124963] Call Trace: [ 3544.125939] [ 3544.126824] ? show_regs.cold.9+0x22/0x2f [ 3544.128440] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3544.130278] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3544.132096] ? watchdog+0x40/0x40 [ 3544.133495] ? hrtimer_interrupt+0x128/0x2c0 [ 3544.135514] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3544.137508] ? apic_timer_interrupt+0xf/0x20 [ 3544.139182] [ 3544.140000] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3544.142244] do_raw_spin_lock+0x177/0x190 [ 3544.143945] _raw_spin_lock+0x21/0x30 [ 3544.145762] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3544.148053] ? do_raw_spin_unlock+0x75/0x190 [ 3544.149756] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3544.152004] ? do_wait_intr+0xf0/0xf0 [ 3544.153625] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3544.156054] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3544.158379] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3544.160863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3544.163098] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3544.165513] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3544.167543] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3544.169340] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3544.171668] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3544.174733] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3544.177086] lmv_intent_open+0x338/0x1150 [lmv] [ 3544.179592] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3544.181747] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3544.184052] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3544.185994] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3544.188099] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3544.190371] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3544.192462] ? do_raw_spin_unlock+0x75/0x190 [ 3544.194298] ? _raw_spin_unlock+0x12/0x30 [ 3544.195915] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3544.198005] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3544.199921] ll_atomic_open+0x36a/0xd70 [lustre] [ 3544.201990] lookup_open+0x356/0xb40 [ 3544.203539] do_last+0x313/0xfc0 [ 3544.204863] ? nd_jump_root+0xe5/0x160 [ 3544.206453] ? path_init+0x437/0x520 [ 3544.207931] path_openat+0xf7/0x500 [ 3544.209491] ? do_raw_spin_unlock+0x75/0x190 [ 3544.211283] do_filp_open+0x99/0x140 [ 3544.212757] ? getname_flags+0x6e/0x330 [ 3544.214477] ? __check_object_size+0xff/0x256 [ 3544.216329] ? do_raw_spin_unlock+0x75/0x190 [ 3544.218226] ? _raw_spin_unlock+0x12/0x30 [ 3544.220251] do_sys_openat2+0x2b4/0x410 [ 3544.221801] do_sys_open+0x73/0xa0 [ 3544.223313] __x64_sys_openat+0x24/0x30 [ 3544.224965] do_syscall_64+0xc1/0x440 [ 3544.226574] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3544.228655] RIP: 0033:0x7fdc0afbf332 [ 3544.230217] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3544.237309] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3544.240177] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3544.242858] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3544.245492] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3544.252657] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3544.256035] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3552.050054] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 3552.064732] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3552.155725] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3552.168131] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3552.191065] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3552.194998] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3552.226000] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3552.234887] RAX: 0000000000007e1a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3552.239452] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3552.246284] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3552.254212] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3552.260147] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3552.264971] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3552.273121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3552.274999] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3552.280412] Call Trace: [ 3552.283981] [ 3552.285902] ? show_regs.cold.9+0x22/0x2f [ 3552.288735] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3552.294574] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3552.296147] ? watchdog+0x40/0x40 [ 3552.302126] ? hrtimer_interrupt+0x128/0x2c0 [ 3552.305155] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3552.311872] ? apic_timer_interrupt+0xf/0x20 [ 3552.317518] [ 3552.318572] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3552.323861] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3552.328937] ? dump_stack+0xf6/0x10e [ 3552.330867] do_raw_spin_lock+0x177/0x190 [ 3552.334914] _raw_spin_lock+0x21/0x30 [ 3552.337952] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3552.340467] obd_mod_free+0x46/0x140 [ptlrpc] [ 3552.346122] mdc_commit_open+0x91/0xb0 [mdc] [ 3552.348406] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3552.352673] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3552.358156] after_reply+0xe2c/0x1780 [ptlrpc] [ 3552.364505] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3552.368555] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3552.370833] ? do_wait_intr+0xf0/0xf0 [ 3552.384185] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3552.386239] kthread+0x1d1/0x200 [ 3552.392499] ? set_kthread_struct+0x70/0x70 [ 3552.396140] ret_from_fork+0x1f/0x30 [ 3556.052794] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3556.057659] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3556.099891] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3556.111669] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3556.115796] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3556.119614] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3556.123740] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3556.132778] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3556.136074] RAX: 0000000000007f88 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3556.142901] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3556.146078] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3556.149176] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3556.152409] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3556.156489] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3556.160040] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3556.162914] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3556.166229] Call Trace: [ 3556.167387] [ 3556.168365] ? show_regs.cold.9+0x22/0x2f [ 3556.170431] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3556.172714] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3556.174654] ? watchdog+0x40/0x40 [ 3556.175970] ? hrtimer_interrupt+0x128/0x2c0 [ 3556.177822] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3556.180414] ? apic_timer_interrupt+0xf/0x20 [ 3556.182479] [ 3556.183378] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3556.186018] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3556.188292] do_raw_spin_lock+0x177/0x190 [ 3556.189811] _raw_spin_lock+0x21/0x30 [ 3556.191550] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3556.194066] process_one_work+0x2c8/0x700 [ 3556.196096] worker_thread+0x296/0x6e0 [ 3556.197661] ? rescuer_thread+0x570/0x570 [ 3556.199653] kthread+0x1d1/0x200 [ 3556.200945] ? set_kthread_struct+0x70/0x70 [ 3556.203060] ret_from_fork+0x1f/0x30 [ 3556.319088] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1531s! [ 3556.322274] Showing busy workqueues and worker pools: [ 3556.326410] workqueue events: flags=0x0 [ 3556.327761] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3556.331954] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3556.336658] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3556.339811] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3556.346676] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3556.349919] pending: kfree_rcu_monitor [ 3556.351178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3556.356112] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3556.360807] workqueue events_long: flags=0x0 [ 3556.362506] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3556.368688] pending: lu_objects_flush [obdclass] [ 3556.371152] workqueue events_unbound: flags=0x2 [ 3556.373531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3556.375630] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3556.382408] workqueue events_power_efficient: flags=0x80 [ 3556.384567] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3556.387891] pending: neigh_periodic_work, check_lifetime [ 3556.390617] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3556.394722] pending: do_cache_clean, neigh_periodic_work [ 3556.407444] workqueue mm_percpu_wq: flags=0x8 [ 3556.412911] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3556.427282] pending: lru_add_drain_per_cpu [ 3556.441495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3556.446777] pending: vmstat_update, lru_add_drain_per_cpu [ 3556.453229] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3556.463075] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3556.469764] workqueue writeback: flags=0x4a [ 3556.473502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3556.476178] pending: wb_workfn, wb_workfn [ 3556.481611] workqueue xprtiod: flags=0xa [ 3556.485706] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3556.492021] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3556.495615] workqueue ipv6_addrconf: flags=0x40008 [ 3556.497667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3556.500736] pending: addrconf_verify_work [ 3556.502414] workqueue knbd0-recv: flags=0x1a [ 3556.503768] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3556.505739] in-flight: 655:recv_work [ 3556.507130] workqueue ptlrpc_pinger: flags=0x2 [ 3556.508687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3556.511147] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3556.514129] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1531s workers=4 idle: 36396 38 11188 [ 3556.516493] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3547s workers=2 idle: 658 [ 3560.056889] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3560.068110] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3560.115707] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3560.123062] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3560.127370] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3560.140538] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3560.166559] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3560.176531] RAX: 000000000000539f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3560.184358] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3560.188567] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3560.201805] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3560.206978] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3560.216072] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3560.223393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3560.227751] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3560.232152] Call Trace: [ 3560.234038] [ 3560.235907] ? show_regs.cold.9+0x22/0x2f [ 3560.238179] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3560.243382] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3560.244824] ? watchdog+0x40/0x40 [ 3560.245735] ? hrtimer_interrupt+0x128/0x2c0 [ 3560.255035] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3560.259727] ? apic_timer_interrupt+0xf/0x20 [ 3560.261975] [ 3560.263860] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3560.267392] do_raw_spin_lock+0x177/0x190 [ 3560.271445] _raw_spin_lock+0x21/0x30 [ 3560.272345] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3560.276918] ? do_raw_spin_unlock+0x75/0x190 [ 3560.278481] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3560.281842] ? do_wait_intr+0xf0/0xf0 [ 3560.283174] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3560.284819] mdc_close+0x53b/0xd70 [mdc] [ 3560.287225] lmv_close+0x18d/0x490 [lmv] [ 3560.288445] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3560.290829] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3560.293737] ll_md_close+0x274/0x800 [lustre] [ 3560.295444] ll_file_release+0x128/0xc50 [lustre] [ 3560.296874] __fput+0x10d/0x440 [ 3560.301570] ____fput+0x12/0x20 [ 3560.302709] task_work_run+0xc6/0x110 [ 3560.303951] exit_to_usermode_loop+0x1dd/0x1f0 [ 3560.306368] do_syscall_64+0x3d6/0x440 [ 3560.307946] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3560.310069] RIP: 0033:0x7fdc0afbfcd8 [ 3560.311392] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3560.317828] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3560.320535] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3560.322623] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3560.325012] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3560.327177] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3560.329432] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3572.052761] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3572.060648] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3572.097962] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3572.105245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3572.111727] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3572.114732] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3572.129148] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3572.133019] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3572.136711] RDX: 00000000000033e2 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3572.139795] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3572.143602] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3572.146878] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3572.149790] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3572.157635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3572.161359] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3572.169121] Call Trace: [ 3572.171247] [ 3572.171833] ? show_regs.cold.9+0x22/0x2f [ 3572.173278] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3572.175721] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3572.177798] ? watchdog+0x40/0x40 [ 3572.180112] ? hrtimer_interrupt+0x128/0x2c0 [ 3572.182460] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3572.184669] ? apic_timer_interrupt+0xf/0x20 [ 3572.186652] [ 3572.187617] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3572.190169] do_raw_spin_lock+0x177/0x190 [ 3572.192165] _raw_spin_lock+0x21/0x30 [ 3572.194043] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3572.199795] ? do_raw_spin_unlock+0x75/0x190 [ 3572.202862] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3572.206287] ? do_wait_intr+0xf0/0xf0 [ 3572.207764] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3572.210244] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3572.213116] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3572.216825] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3572.219809] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3572.222268] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3572.224084] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3572.226363] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3572.228837] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3572.232201] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3572.234648] lmv_intent_open+0x338/0x1150 [lmv] [ 3572.236749] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3572.238792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3572.241680] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3572.244146] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3572.246437] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3572.249221] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3572.251413] ? do_raw_spin_unlock+0x75/0x190 [ 3572.253675] ? _raw_spin_unlock+0x12/0x30 [ 3572.255361] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3572.257797] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3572.260178] ll_atomic_open+0x36a/0xd70 [lustre] [ 3572.262706] lookup_open+0x356/0xb40 [ 3572.264867] do_last+0x313/0xfc0 [ 3572.266697] ? nd_jump_root+0xe5/0x160 [ 3572.268742] ? path_init+0x437/0x520 [ 3572.270768] path_openat+0xf7/0x500 [ 3572.272246] ? do_raw_spin_unlock+0x75/0x190 [ 3572.274164] do_filp_open+0x99/0x140 [ 3572.275767] ? getname_flags+0x6e/0x330 [ 3572.277436] ? __check_object_size+0xff/0x256 [ 3572.279369] ? do_raw_spin_unlock+0x75/0x190 [ 3572.281401] ? _raw_spin_unlock+0x12/0x30 [ 3572.283344] do_sys_openat2+0x2b4/0x410 [ 3572.285095] do_sys_open+0x73/0xa0 [ 3572.286733] __x64_sys_openat+0x24/0x30 [ 3572.288416] do_syscall_64+0xc1/0x440 [ 3572.290190] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3572.292348] RIP: 0033:0x7fdc0afbf332 [ 3572.294079] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3572.303467] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3572.306811] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3572.309269] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3572.312446] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3572.315454] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3572.318343] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3584.048016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3584.052250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3584.057031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3584.073515] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3584.073535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3584.073538] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3584.073551] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3584.073555] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3584.073564] RAX: 0000000000006b88 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3584.073566] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3584.073571] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3584.073573] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3584.073576] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3584.073579] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3584.073582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3584.073584] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3584.073591] Call Trace: [ 3584.073596] [ 3584.073602] ? show_regs.cold.9+0x22/0x2f [ 3584.073610] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3584.073617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3584.073622] ? watchdog+0x40/0x40 [ 3584.073626] ? hrtimer_interrupt+0x128/0x2c0 [ 3584.073747] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3584.073757] ? apic_timer_interrupt+0xf/0x20 [ 3584.073762] [ 3584.073765] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3584.073770] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3584.073773] ? dump_stack+0xf6/0x10e [ 3584.073779] do_raw_spin_lock+0x177/0x190 [ 3584.073784] _raw_spin_lock+0x21/0x30 [ 3584.073791] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3584.074540] obd_mod_free+0x46/0x140 [ptlrpc] [ 3584.074819] mdc_commit_open+0x91/0xb0 [mdc] [ 3584.074858] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3584.080037] Modules linked in: lustre(O) [ 3584.085965] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3584.086437] after_reply+0xe2c/0x1780 [ptlrpc] [ 3584.086672] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3584.087085] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3584.087412] ? do_wait_intr+0xf0/0xf0 [ 3584.087422] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3584.099400] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3584.099486] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3584.099496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3584.099503] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3584.100024] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3584.100037] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3584.100057] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 [ 3584.105387] kthread+0x1d1/0x200 [ 3584.105405] ? set_kthread_struct+0x70/0x70 [ 3584.105410] ret_from_fork+0x1f/0x30 [ 3584.251498] ORIG_RAX: ffffffffffffff13 [ 3584.252644] RAX: 0000000000007d8c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3584.255055] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3584.257353] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3584.259453] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3584.262028] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3584.264992] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3584.270991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3584.273868] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3584.276578] Call Trace: [ 3584.277509] [ 3584.278235] ? show_regs.cold.9+0x22/0x2f [ 3584.279579] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3584.281601] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3584.283739] ? watchdog+0x40/0x40 [ 3584.285376] ? hrtimer_interrupt+0x128/0x2c0 [ 3584.287080] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3584.289147] ? apic_timer_interrupt+0xf/0x20 [ 3584.290791] [ 3584.291515] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3584.293615] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3584.295474] do_raw_spin_lock+0x177/0x190 [ 3584.296974] _raw_spin_lock+0x21/0x30 [ 3584.298551] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3584.301108] process_one_work+0x2c8/0x700 [ 3584.302780] worker_thread+0x296/0x6e0 [ 3584.304465] ? rescuer_thread+0x570/0x570 [ 3584.306102] kthread+0x1d1/0x200 [ 3584.307771] ? set_kthread_struct+0x70/0x70 [ 3584.310002] ret_from_fork+0x1f/0x30 [ 3586.527372] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1561s! [ 3586.533651] Showing busy workqueues and worker pools: [ 3586.537026] workqueue events: flags=0x0 [ 3586.538194] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3586.544313] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3586.550039] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3586.558578] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3586.563767] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3586.568429] pending: kfree_rcu_monitor [ 3586.571960] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3586.575904] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3586.581069] workqueue events_long: flags=0x0 [ 3586.582896] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3586.585560] pending: lu_objects_flush [obdclass] [ 3586.590834] workqueue events_unbound: flags=0x2 [ 3586.592260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3586.595524] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3586.599856] workqueue events_power_efficient: flags=0x80 [ 3586.602756] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3586.606069] pending: neigh_periodic_work, check_lifetime [ 3586.608326] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3586.610734] pending: do_cache_clean, neigh_periodic_work [ 3586.613013] workqueue mm_percpu_wq: flags=0x8 [ 3586.615888] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3586.618510] pending: lru_add_drain_per_cpu [ 3586.620500] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3586.623141] pending: vmstat_update, lru_add_drain_per_cpu [ 3586.626881] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3586.632382] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3586.634223] workqueue writeback: flags=0x4a [ 3586.635503] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3586.638419] pending: wb_workfn, wb_workfn [ 3586.642042] workqueue xprtiod: flags=0xa [ 3586.643723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3586.645888] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3586.648320] workqueue ipv6_addrconf: flags=0x40008 [ 3586.649987] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3586.652186] pending: addrconf_verify_work [ 3586.656488] workqueue knbd0-recv: flags=0x1a [ 3586.658594] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3586.664557] in-flight: 655:recv_work [ 3586.666728] workqueue ptlrpc_pinger: flags=0x2 [ 3586.669403] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3586.672273] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3586.676196] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1561s workers=4 idle: 36396 38 11188 [ 3586.679718] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3577s workers=2 idle: 658 [ 3588.054649] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3588.058797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3588.091432] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3588.099035] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3588.103518] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3588.108329] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3588.122773] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3588.126001] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3588.130368] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3588.136430] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3588.138645] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3588.141350] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3588.144599] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3588.153778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3588.157804] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3588.168046] Call Trace: [ 3588.168898] [ 3588.172304] ? show_regs.cold.9+0x22/0x2f [ 3588.176883] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3588.180164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3588.185527] ? watchdog+0x40/0x40 [ 3588.189386] ? hrtimer_interrupt+0x128/0x2c0 [ 3588.192345] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3588.196990] ? apic_timer_interrupt+0xf/0x20 [ 3588.200560] [ 3588.202167] ? native_safe_halt+0x2e/0x50 [ 3588.204154] kvm_wait+0xcd/0xe0 [ 3588.206383] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3588.208580] do_raw_spin_lock+0x177/0x190 [ 3588.211402] _raw_spin_lock+0x21/0x30 [ 3588.213166] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3588.219587] ? do_raw_spin_unlock+0x75/0x190 [ 3588.227524] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3588.231767] ? do_wait_intr+0xf0/0xf0 [ 3588.234679] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3588.243483] mdc_close+0x53b/0xd70 [mdc] [ 3588.247780] lmv_close+0x18d/0x490 [lmv] [ 3588.252230] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3588.256833] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3588.260090] ll_md_close+0x274/0x800 [lustre] [ 3588.262677] ll_file_release+0x128/0xc50 [lustre] [ 3588.267112] __fput+0x10d/0x440 [ 3588.273001] ____fput+0x12/0x20 [ 3588.276004] task_work_run+0xc6/0x110 [ 3588.277208] exit_to_usermode_loop+0x1dd/0x1f0 [ 3588.278427] do_syscall_64+0x3d6/0x440 [ 3588.280138] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3588.282533] RIP: 0033:0x7fdc0afbfcd8 [ 3588.283994] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3588.292074] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3588.295419] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3588.298401] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3588.301354] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3588.304128] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3588.307404] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3600.052008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3600.057127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3600.102559] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3600.110346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3600.114697] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3600.117883] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3600.126191] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3600.129857] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3600.133007] RDX: 0000000000002549 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3600.136772] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3600.140330] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3600.144356] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3600.147734] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3600.152698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3600.164710] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3600.171239] Call Trace: [ 3600.172366] [ 3600.176145] ? show_regs.cold.9+0x22/0x2f [ 3600.181394] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3600.187895] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3600.190056] ? watchdog+0x40/0x40 [ 3600.194569] ? hrtimer_interrupt+0x128/0x2c0 [ 3600.196565] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3600.201830] ? apic_timer_interrupt+0xf/0x20 [ 3600.203316] [ 3600.204839] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3600.208207] do_raw_spin_lock+0x177/0x190 [ 3600.211303] _raw_spin_lock+0x21/0x30 [ 3600.214327] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3600.217344] ? do_raw_spin_unlock+0x75/0x190 [ 3600.219765] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3600.230842] ? do_wait_intr+0xf0/0xf0 [ 3600.238201] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3600.242656] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3600.246249] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3600.251180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3600.256003] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3600.261307] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3600.266874] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3600.270930] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3600.273897] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3600.277697] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3600.280846] lmv_intent_open+0x338/0x1150 [lmv] [ 3600.283335] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3600.286772] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3600.290117] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3600.292939] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3600.295101] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3600.301140] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3600.305285] ? do_raw_spin_unlock+0x75/0x190 [ 3600.307528] ? _raw_spin_unlock+0x12/0x30 [ 3600.309890] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3600.311871] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3600.316009] ll_atomic_open+0x36a/0xd70 [lustre] [ 3600.321792] lookup_open+0x356/0xb40 [ 3600.323792] do_last+0x313/0xfc0 [ 3600.325396] ? nd_jump_root+0xe5/0x160 [ 3600.327471] ? path_init+0x437/0x520 [ 3600.329220] path_openat+0xf7/0x500 [ 3600.331731] ? do_raw_spin_unlock+0x75/0x190 [ 3600.336680] do_filp_open+0x99/0x140 [ 3600.339195] ? getname_flags+0x6e/0x330 [ 3600.341169] ? __check_object_size+0xff/0x256 [ 3600.346035] ? do_raw_spin_unlock+0x75/0x190 [ 3600.350071] ? _raw_spin_unlock+0x12/0x30 [ 3600.353720] do_sys_openat2+0x2b4/0x410 [ 3600.360054] do_sys_open+0x73/0xa0 [ 3600.362178] __x64_sys_openat+0x24/0x30 [ 3600.365582] do_syscall_64+0xc1/0x440 [ 3600.368670] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3600.370880] RIP: 0033:0x7fdc0afbf332 [ 3600.371888] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3600.382475] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3600.386528] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3600.390531] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3600.394326] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3600.397794] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3600.400322] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3612.048129] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3612.051024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3612.052815] Modules linked in: lustre(O) [ 3612.056003] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [ 3612.058188] osp(O) [ 3612.068718] intel_rapl_msr [ 3612.068721] ofd(O) [ 3612.068729] intel_rapl_common sb_edac [ 3612.070070] lod(O) [ 3612.071445] rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul [ 3612.072690] mdt(O) [ 3612.074564] crc32c_intel [ 3612.075759] mdd(O) [ 3612.079815] ata_piix ghash_clmulni_intel [ 3612.081176] mgs(O) [ 3612.082467] libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3612.082485] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3612.082491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3612.082498] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3612.083652] lquota(O) [ 3612.085475] [ 3612.085486] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3612.085505] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3612.085508] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3612.085513] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3612.085515] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3612.085517] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3612.085519] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3612.085521] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3612.085523] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3612.085526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3612.085528] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3612.085535] Call Trace: [ 3612.085541] [ 3612.085545] ? show_regs.cold.9+0x22/0x2f [ 3612.085554] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3612.085560] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3612.085564] ? watchdog+0x40/0x40 [ 3612.085568] ? hrtimer_interrupt+0x128/0x2c0 [ 3612.085572] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3612.085709] ? apic_timer_interrupt+0xf/0x20 [ 3612.085746] [ 3612.085749] ? native_safe_halt+0x2e/0x50 [ 3612.085753] kvm_wait+0xcd/0xe0 [ 3612.085761] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3612.085771] do_raw_spin_lock+0x177/0x190 [ 3612.085775] _raw_spin_lock+0x21/0x30 [ 3612.085784] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3612.086269] process_one_work+0x2c8/0x700 [ 3612.086289] worker_thread+0x296/0x6e0 [ 3612.086294] ? rescuer_thread+0x570/0x570 [ 3612.086297] kthread+0x1d1/0x200 [ 3612.086301] ? set_kthread_struct+0x70/0x70 [ 3612.086305] ret_from_fork+0x1f/0x30 [ 3612.177421] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3612.190124] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3612.193317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3612.196005] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3612.198051] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3612.204491] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3612.206935] RAX: 000000000000720f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3612.209040] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3612.211289] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3612.216160] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3612.219809] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3612.223321] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3612.227038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3612.229294] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3612.232161] Call Trace: [ 3612.233266] [ 3612.234071] ? show_regs.cold.9+0x22/0x2f [ 3612.235818] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3612.237736] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3612.239730] ? watchdog+0x40/0x40 [ 3612.241127] ? hrtimer_interrupt+0x128/0x2c0 [ 3612.242995] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3612.245003] ? apic_timer_interrupt+0xf/0x20 [ 3612.246890] [ 3612.247885] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3612.250590] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3612.254079] ? dump_stack+0xf6/0x10e [ 3612.255439] do_raw_spin_lock+0x177/0x190 [ 3612.257151] _raw_spin_lock+0x21/0x30 [ 3612.258788] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3612.261124] obd_mod_free+0x46/0x140 [ptlrpc] [ 3612.263977] mdc_commit_open+0x91/0xb0 [mdc] [ 3612.265930] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3612.268445] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3612.271131] after_reply+0xe2c/0x1780 [ptlrpc] [ 3612.273306] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3612.275866] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3612.277920] ? do_wait_intr+0xf0/0xf0 [ 3612.279457] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3612.282550] kthread+0x1d1/0x200 [ 3612.284079] ? set_kthread_struct+0x70/0x70 [ 3612.285841] ret_from_fork+0x1f/0x30 [ 3616.062577] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3616.068779] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3616.108795] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3616.122744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3616.130463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3616.139175] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3616.165634] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3616.172332] RAX: 0000000000007b82 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3616.178023] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3616.184404] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3616.188947] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3616.195540] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3616.200730] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3616.209064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3616.216083] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3616.227567] Call Trace: [ 3616.228443] [ 3616.230688] ? show_regs.cold.9+0x22/0x2f [ 3616.232219] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3616.234823] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3616.237863] ? watchdog+0x40/0x40 [ 3616.241693] ? hrtimer_interrupt+0x128/0x2c0 [ 3616.245962] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3616.248501] ? apic_timer_interrupt+0xf/0x20 [ 3616.254015] [ 3616.255278] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3616.265737] do_raw_spin_lock+0x177/0x190 [ 3616.267486] _raw_spin_lock+0x21/0x30 [ 3616.268985] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3616.271723] ? do_raw_spin_unlock+0x75/0x190 [ 3616.276095] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3616.280690] ? do_wait_intr+0xf0/0xf0 [ 3616.282570] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3616.285098] mdc_close+0x53b/0xd70 [mdc] [ 3616.287466] lmv_close+0x18d/0x490 [lmv] [ 3616.290075] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3616.292737] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3616.296083] ll_md_close+0x274/0x800 [lustre] [ 3616.298349] ll_file_release+0x128/0xc50 [lustre] [ 3616.301423] __fput+0x10d/0x440 [ 3616.302733] ____fput+0x12/0x20 [ 3616.305963] task_work_run+0xc6/0x110 [ 3616.307965] exit_to_usermode_loop+0x1dd/0x1f0 [ 3616.312054] do_syscall_64+0x3d6/0x440 [ 3616.313413] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3616.316605] RIP: 0033:0x7fdc0afbfcd8 [ 3616.317717] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3616.326357] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3616.329041] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3616.332432] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3616.339115] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3616.342905] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3616.349764] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3616.735687] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1591s! [ 3616.745717] Showing busy workqueues and worker pools: [ 3616.748425] workqueue events: flags=0x0 [ 3616.750713] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3616.758361] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3616.761993] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3616.768217] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3616.774199] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3616.777112] pending: kfree_rcu_monitor [ 3616.781865] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3616.785663] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3616.790802] workqueue events_long: flags=0x0 [ 3616.793238] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3616.799321] pending: lu_objects_flush [obdclass] [ 3616.801877] workqueue events_unbound: flags=0x2 [ 3616.803700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3616.808490] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3616.814116] workqueue events_power_efficient: flags=0x80 [ 3616.818072] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3616.821980] pending: neigh_periodic_work, check_lifetime [ 3616.825100] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3616.830064] pending: do_cache_clean, neigh_periodic_work [ 3616.833225] workqueue mm_percpu_wq: flags=0x8 [ 3616.836321] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3616.839697] pending: lru_add_drain_per_cpu [ 3616.842158] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3616.846451] pending: vmstat_update, lru_add_drain_per_cpu [ 3616.849484] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3616.852948] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3616.857420] workqueue writeback: flags=0x4a [ 3616.859761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3616.866499] pending: wb_workfn, wb_workfn [ 3616.868420] workqueue xprtiod: flags=0xa [ 3616.869941] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3616.872957] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3616.878725] workqueue ipv6_addrconf: flags=0x40008 [ 3616.881180] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3616.884455] pending: addrconf_verify_work [ 3616.886483] workqueue knbd0-recv: flags=0x1a [ 3616.887628] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3616.894094] in-flight: 655:recv_work [ 3616.896650] workqueue ptlrpc_pinger: flags=0x2 [ 3616.898564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3616.901912] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3616.904721] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1591s workers=4 idle: 36396 38 11188 [ 3616.908068] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3607s workers=2 idle: 658 [ 3628.052264] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3628.067168] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3628.106295] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3628.110554] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3628.114023] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3628.116235] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3628.123284] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3628.126256] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3628.129130] RDX: 0000000000003a57 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3628.132238] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3628.135145] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3628.138137] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3628.141042] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3628.144178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3628.146589] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3628.149752] Call Trace: [ 3628.150851] [ 3628.151719] ? show_regs.cold.9+0x22/0x2f [ 3628.153467] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3628.155690] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3628.157467] ? watchdog+0x40/0x40 [ 3628.159126] ? hrtimer_interrupt+0x128/0x2c0 [ 3628.160988] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3628.163038] ? apic_timer_interrupt+0xf/0x20 [ 3628.164768] [ 3628.165746] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3628.168113] do_raw_spin_lock+0x177/0x190 [ 3628.169845] _raw_spin_lock+0x21/0x30 [ 3628.171340] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3628.173978] ? do_raw_spin_unlock+0x75/0x190 [ 3628.175571] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3628.178332] ? do_wait_intr+0xf0/0xf0 [ 3628.180283] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3628.183034] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3628.185556] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3628.188513] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3628.192217] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3628.194580] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3628.197863] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3628.200829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3628.202258] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3628.207912] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3628.209900] lmv_intent_open+0x338/0x1150 [lmv] [ 3628.211529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3628.215538] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3628.217337] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3628.219227] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3628.221230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3628.223846] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3628.226036] ? do_raw_spin_unlock+0x75/0x190 [ 3628.227797] ? _raw_spin_unlock+0x12/0x30 [ 3628.229531] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3628.232337] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3628.234588] ll_atomic_open+0x36a/0xd70 [lustre] [ 3628.237253] lookup_open+0x356/0xb40 [ 3628.238854] do_last+0x313/0xfc0 [ 3628.240691] ? nd_jump_root+0xe5/0x160 [ 3628.242370] ? path_init+0x437/0x520 [ 3628.244125] path_openat+0xf7/0x500 [ 3628.245993] ? do_raw_spin_unlock+0x75/0x190 [ 3628.248002] do_filp_open+0x99/0x140 [ 3628.249837] ? getname_flags+0x6e/0x330 [ 3628.251558] ? __check_object_size+0xff/0x256 [ 3628.253467] ? do_raw_spin_unlock+0x75/0x190 [ 3628.255959] ? _raw_spin_unlock+0x12/0x30 [ 3628.257650] do_sys_openat2+0x2b4/0x410 [ 3628.259489] do_sys_open+0x73/0xa0 [ 3628.261140] __x64_sys_openat+0x24/0x30 [ 3628.262863] do_syscall_64+0xc1/0x440 [ 3628.264493] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3628.267021] RIP: 0033:0x7fdc0afbf332 [ 3628.268490] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3628.275549] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3628.278356] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3628.281690] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3628.284554] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3628.288232] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3628.291269] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3640.051338] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 3640.053939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3640.070164] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3640.073503] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3640.076190] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3640.078231] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3640.079587] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3640.084540] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3640.086587] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3640.088764] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3640.090619] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3640.093217] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3640.095672] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3640.100780] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3640.105668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3640.119317] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3640.126812] Call Trace: [ 3640.127596] [ 3640.128177] ? show_regs.cold.9+0x22/0x2f [ 3640.129291] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3640.148393] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3640.149470] ? watchdog+0x40/0x40 [ 3640.150423] ? hrtimer_interrupt+0x128/0x2c0 [ 3640.157597] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3640.163770] ? apic_timer_interrupt+0xf/0x20 [ 3640.166375] [ 3640.170469] ? native_safe_halt+0x2e/0x50 [ 3640.171517] kvm_wait+0xcd/0xe0 [ 3640.172359] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3640.176347] do_raw_spin_lock+0x177/0x190 [ 3640.178238] _raw_spin_lock+0x21/0x30 [ 3640.180434] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3640.183237] process_one_work+0x2c8/0x700 [ 3640.185256] worker_thread+0x296/0x6e0 [ 3640.187285] ? rescuer_thread+0x570/0x570 [ 3640.189002] kthread+0x1d1/0x200 [ 3640.192462] ? set_kthread_struct+0x70/0x70 [ 3640.195488] ret_from_fork+0x1f/0x30 [ 3644.049259] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3644.054027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3644.055776] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 3644.059599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3644.059852] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3644.059859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3644.059863] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3644.059875] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3644.059879] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3644.059884] RAX: 0000000000005882 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3644.059886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3644.059888] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3644.059890] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3644.059893] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3644.059896] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3644.059899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3644.059901] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3644.059911] Call Trace: [ 3644.059916] [ 3644.059921] ? show_regs.cold.9+0x22/0x2f [ 3644.059930] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3644.059936] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3644.059941] ? watchdog+0x40/0x40 [ 3644.059945] ? hrtimer_interrupt+0x128/0x2c0 [ 3644.059949] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3644.059957] ? apic_timer_interrupt+0xf/0x20 [ 3644.059962] [ 3644.059965] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3644.059972] do_raw_spin_lock+0x177/0x190 [ 3644.059976] _raw_spin_lock+0x21/0x30 [ 3644.059981] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3644.083220] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3644.083263] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3644.083271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3644.083273] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3644.083286] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3644.083437] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3644.083441] RAX: 0000000000003f07 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3644.083444] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3644.083446] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3644.083448] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3644.083450] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3644.083453] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3644.083456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3644.083458] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3644.083466] Call Trace: [ 3644.083471] [ 3644.083477] ? show_regs.cold.9+0x22/0x2f [ 3644.083486] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3644.083492] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3644.083496] ? watchdog+0x40/0x40 [ 3644.083500] ? hrtimer_interrupt+0x128/0x2c0 [ 3644.083503] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3644.083511] ? apic_timer_interrupt+0xf/0x20 [ 3644.083516] [ 3644.083520] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3644.083524] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3644.083527] ? dump_stack+0xf6/0x10e [ 3644.083533] do_raw_spin_lock+0x177/0x190 [ 3644.083537] _raw_spin_lock+0x21/0x30 [ 3644.083544] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3644.083804] obd_mod_free+0x46/0x140 [ptlrpc] [ 3644.083995] mdc_commit_open+0x91/0xb0 [mdc] [ 3644.084031] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3644.084709] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3644.084906] after_reply+0xe2c/0x1780 [ptlrpc] [ 3644.085227] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3644.090436] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3644.091185] ? do_wait_intr+0xf0/0xf0 [ 3644.091195] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3644.092075] kthread+0x1d1/0x200 [ 3644.092081] ? set_kthread_struct+0x70/0x70 [ 3644.092085] ret_from_fork+0x1f/0x30 [ 3644.723288] ? do_raw_spin_unlock+0x75/0x190 [ 3644.727794] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3644.737015] ? do_wait_intr+0xf0/0xf0 [ 3644.740151] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3644.745925] mdc_close+0x53b/0xd70 [mdc] [ 3644.747288] lmv_close+0x18d/0x490 [lmv] [ 3644.752104] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3644.756773] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3644.759961] ll_md_close+0x274/0x800 [lustre] [ 3644.762855] ll_file_release+0x128/0xc50 [lustre] [ 3644.767013] __fput+0x10d/0x440 [ 3644.770312] ____fput+0x12/0x20 [ 3644.773384] task_work_run+0xc6/0x110 [ 3644.777181] exit_to_usermode_loop+0x1dd/0x1f0 [ 3644.782375] do_syscall_64+0x3d6/0x440 [ 3644.791219] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3644.796996] RIP: 0033:0x7fdc0afbfcd8 [ 3644.802569] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3644.815751] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3644.821018] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3644.825542] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3644.831924] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3644.841417] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3644.849104] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3646.946027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1622s! [ 3646.954631] Showing busy workqueues and worker pools: [ 3646.958014] workqueue events: flags=0x0 [ 3646.959907] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3646.964480] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3646.967359] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3646.974701] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3646.985017] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3646.988096] pending: kfree_rcu_monitor [ 3646.991276] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3646.996808] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 3647.002612] workqueue events_long: flags=0x0 [ 3647.004211] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3647.007577] pending: lu_objects_flush [obdclass] [ 3647.010200] workqueue events_unbound: flags=0x2 [ 3647.015351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3647.017985] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3647.027932] workqueue events_power_efficient: flags=0x80 [ 3647.033295] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3647.037229] pending: neigh_periodic_work, check_lifetime [ 3647.038652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3647.045670] pending: do_cache_clean, neigh_periodic_work [ 3647.049806] workqueue mm_percpu_wq: flags=0x8 [ 3647.051272] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3647.056731] pending: lru_add_drain_per_cpu [ 3647.062236] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3647.066967] pending: vmstat_update, lru_add_drain_per_cpu [ 3647.072794] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3647.082086] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3647.085304] workqueue writeback: flags=0x4a [ 3647.088112] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3647.092438] pending: wb_workfn, wb_workfn [ 3647.096260] workqueue xprtiod: flags=0xa [ 3647.098559] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3647.107231] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3647.109279] workqueue ipv6_addrconf: flags=0x40008 [ 3647.111826] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3647.117077] pending: addrconf_verify_work [ 3647.120760] workqueue knbd0-recv: flags=0x1a [ 3647.121881] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3647.124611] in-flight: 655:recv_work [ 3647.127418] workqueue ptlrpc_pinger: flags=0x2 [ 3647.130196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3647.136607] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3647.141907] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1622s workers=4 idle: 36396 38 11188 [ 3647.145630] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3637s workers=2 idle: 658 [ 3656.052029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3656.054829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3656.078572] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3656.083963] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3656.090651] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3656.104613] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3656.128134] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3656.133182] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3656.136231] RDX: 00000000000037e7 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3656.146472] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3656.163320] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3656.167973] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3656.172517] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3656.183354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3656.194106] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3656.204151] Call Trace: [ 3656.208288] [ 3656.209235] ? show_regs.cold.9+0x22/0x2f [ 3656.211103] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3656.214092] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3656.219999] ? watchdog+0x40/0x40 [ 3656.223955] ? hrtimer_interrupt+0x128/0x2c0 [ 3656.231962] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3656.236233] ? apic_timer_interrupt+0xf/0x20 [ 3656.243221] [ 3656.246152] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3656.254041] do_raw_spin_lock+0x177/0x190 [ 3656.258909] _raw_spin_lock+0x21/0x30 [ 3656.262093] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3656.268232] ? do_raw_spin_unlock+0x75/0x190 [ 3656.271933] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3656.282046] ? do_wait_intr+0xf0/0xf0 [ 3656.287283] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3656.297322] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3656.300335] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3656.306366] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3656.312209] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3656.314353] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3656.318511] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3656.321324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3656.332484] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3656.341569] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3656.347140] lmv_intent_open+0x338/0x1150 [lmv] [ 3656.349246] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3656.355974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3656.361176] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3656.363977] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3656.366317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3656.374927] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3656.379029] ? do_raw_spin_unlock+0x75/0x190 [ 3656.385236] ? _raw_spin_unlock+0x12/0x30 [ 3656.390070] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3656.393160] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3656.397801] ll_atomic_open+0x36a/0xd70 [lustre] [ 3656.403003] lookup_open+0x356/0xb40 [ 3656.404982] do_last+0x313/0xfc0 [ 3656.408166] ? nd_jump_root+0xe5/0x160 [ 3656.410160] ? path_init+0x437/0x520 [ 3656.415968] path_openat+0xf7/0x500 [ 3656.418313] ? do_raw_spin_unlock+0x75/0x190 [ 3656.426488] do_filp_open+0x99/0x140 [ 3656.429223] ? getname_flags+0x6e/0x330 [ 3656.433352] ? __check_object_size+0xff/0x256 [ 3656.439002] ? do_raw_spin_unlock+0x75/0x190 [ 3656.441928] ? _raw_spin_unlock+0x12/0x30 [ 3656.444480] do_sys_openat2+0x2b4/0x410 [ 3656.446932] do_sys_open+0x73/0xa0 [ 3656.449065] __x64_sys_openat+0x24/0x30 [ 3656.451194] do_syscall_64+0xc1/0x440 [ 3656.453231] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3656.455814] RIP: 0033:0x7fdc0afbf332 [ 3656.458056] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3656.468791] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3656.472295] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3656.475419] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3656.479125] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3656.491486] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3656.497296] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3668.051028] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 3668.053485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3668.080936] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3668.086936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3668.095563] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3668.098480] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3668.099641] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3668.111946] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3668.118991] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3668.122877] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3668.126614] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3668.132726] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3668.134734] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3668.137920] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3668.142432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3668.145281] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3668.148693] Call Trace: [ 3668.151322] [ 3668.153379] ? show_regs.cold.9+0x22/0x2f [ 3668.155286] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3668.157663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3668.161533] ? watchdog+0x40/0x40 [ 3668.164864] ? hrtimer_interrupt+0x128/0x2c0 [ 3668.168966] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3668.172452] ? apic_timer_interrupt+0xf/0x20 [ 3668.174320] [ 3668.174972] ? native_safe_halt+0x2e/0x50 [ 3668.179897] kvm_wait+0xcd/0xe0 [ 3668.182361] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3668.184436] do_raw_spin_lock+0x177/0x190 [ 3668.187448] _raw_spin_lock+0x21/0x30 [ 3668.197115] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3668.198985] process_one_work+0x2c8/0x700 [ 3668.208676] worker_thread+0x296/0x6e0 [ 3668.211643] ? rescuer_thread+0x570/0x570 [ 3668.217326] kthread+0x1d1/0x200 [ 3668.220682] ? set_kthread_struct+0x70/0x70 [ 3668.225114] ret_from_fork+0x1f/0x30 [ 3672.048020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3672.049942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3672.054018] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3672.061573] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3672.064500] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3672.064586] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3672.064594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3672.064597] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3672.064611] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3672.064616] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3672.064620] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3672.064623] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3672.064625] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3672.064627] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3672.064630] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3672.064633] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3672.064636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3672.064638] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3672.064648] Call Trace: [ 3672.064653] [ 3672.064658] ? show_regs.cold.9+0x22/0x2f [ 3672.064667] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3672.064675] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3672.064679] ? watchdog+0x40/0x40 [ 3672.064684] ? hrtimer_interrupt+0x128/0x2c0 [ 3672.064688] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3672.064694] ? apic_timer_interrupt+0xf/0x20 [ 3672.064699] [ 3672.064702] ? native_safe_halt+0x2e/0x50 [ 3672.064706] kvm_wait+0xcd/0xe0 [ 3672.064714] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3672.064722] do_raw_spin_lock+0x177/0x190 [ 3672.064727] _raw_spin_lock+0x21/0x30 [ 3672.064730] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3672.064998] ? do_raw_spin_unlock+0x75/0x190 [ 3672.065004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3672.065202] ? do_wait_intr+0xf0/0xf0 [ 3672.065209] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3672.065406] mdc_close+0x53b/0xd70 [mdc] [ 3672.065445] lmv_close+0x18d/0x490 [lmv] [ 3672.065475] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3672.065619] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3672.065700] ll_md_close+0x274/0x800 [lustre] [ 3672.065780] ll_file_release+0x128/0xc50 [lustre] [ 3672.065861] __fput+0x10d/0x440 [ 3672.065870] ____fput+0x12/0x20 [ 3672.065874] task_work_run+0xc6/0x110 [ 3672.065881] exit_to_usermode_loop+0x1dd/0x1f0 [ 3672.065888] do_syscall_64+0x3d6/0x440 [ 3672.065894] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3672.065900] RIP: 0033:0x7fdc0afbfcd8 [ 3672.065907] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3672.065911] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3672.065915] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3672.065917] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3672.065919] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3672.065921] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3672.065922] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3672.247113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3672.255637] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3672.260454] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3672.276274] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3672.284953] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3672.289488] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3672.293567] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3672.298666] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3672.308427] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3672.313925] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3672.319195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3672.321218] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3672.328434] Call Trace: [ 3672.329712] [ 3672.330936] ? show_regs.cold.9+0x22/0x2f [ 3672.333282] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3672.336460] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3672.339493] ? watchdog+0x40/0x40 [ 3672.342588] ? hrtimer_interrupt+0x128/0x2c0 [ 3672.344615] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3672.351497] ? apic_timer_interrupt+0xf/0x20 [ 3672.360184] [ 3672.363237] ? native_safe_halt+0x2e/0x50 [ 3672.367483] kvm_wait+0xcd/0xe0 [ 3672.368219] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3672.369439] ? dump_stack+0xf6/0x10e [ 3672.379096] do_raw_spin_lock+0x177/0x190 [ 3672.382016] _raw_spin_lock+0x21/0x30 [ 3672.384696] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3672.386332] obd_mod_free+0x46/0x140 [ptlrpc] [ 3672.388720] mdc_commit_open+0x91/0xb0 [mdc] [ 3672.390497] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3672.397797] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3672.401394] after_reply+0xe2c/0x1780 [ptlrpc] [ 3672.406747] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3672.410579] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3672.419106] ? do_wait_intr+0xf0/0xf0 [ 3672.421950] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3672.426492] kthread+0x1d1/0x200 [ 3672.428433] ? set_kthread_struct+0x70/0x70 [ 3672.430533] ret_from_fork+0x1f/0x30 [ 3677.151459] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1652s! [ 3677.155460] Showing busy workqueues and worker pools: [ 3677.157747] workqueue events: flags=0x0 [ 3677.159705] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3677.163501] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3677.166798] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3677.169793] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3677.175412] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3677.178674] pending: kfree_rcu_monitor [ 3677.180673] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3677.183509] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 3677.188929] workqueue events_long: flags=0x0 [ 3677.190963] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3677.193932] pending: lu_objects_flush [obdclass] [ 3677.196444] workqueue events_unbound: flags=0x2 [ 3677.198620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3677.201524] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3677.205727] workqueue events_power_efficient: flags=0x80 [ 3677.207980] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3677.211602] pending: neigh_periodic_work, check_lifetime [ 3677.213595] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3677.217400] pending: do_cache_clean, neigh_periodic_work [ 3677.220659] workqueue mm_percpu_wq: flags=0x8 [ 3677.222726] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3677.226246] pending: lru_add_drain_per_cpu [ 3677.228514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3677.231768] pending: vmstat_update, lru_add_drain_per_cpu [ 3677.235132] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3677.238302] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3677.241530] workqueue writeback: flags=0x4a [ 3677.247678] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3677.252689] pending: wb_workfn, wb_workfn [ 3677.254989] workqueue xprtiod: flags=0xa [ 3677.257109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3677.260001] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3677.263292] workqueue ipv6_addrconf: flags=0x40008 [ 3677.265051] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3677.267427] pending: addrconf_verify_work [ 3677.269289] workqueue knbd0-recv: flags=0x1a [ 3677.270733] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3677.273583] in-flight: 655:recv_work [ 3677.275554] workqueue ptlrpc_pinger: flags=0x2 [ 3677.282164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3677.286385] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3677.288443] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1652s workers=4 idle: 36396 38 11188 [ 3677.291596] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3667s workers=2 idle: 658 [ 3684.056150] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 3684.062769] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3684.114380] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3684.120706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3684.127206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3684.134319] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3684.153631] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3684.161841] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3684.164145] RDX: 000000000000281c RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3684.172748] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3684.176402] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3684.179764] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3684.182858] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3684.186525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3684.188141] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3684.190070] Call Trace: [ 3684.191741] [ 3684.192673] ? show_regs.cold.9+0x22/0x2f [ 3684.194628] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3684.197322] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3684.200495] ? watchdog+0x40/0x40 [ 3684.202463] ? hrtimer_interrupt+0x128/0x2c0 [ 3684.204645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3684.207470] ? apic_timer_interrupt+0xf/0x20 [ 3684.209597] [ 3684.210644] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3684.213301] do_raw_spin_lock+0x177/0x190 [ 3684.215258] _raw_spin_lock+0x21/0x30 [ 3684.216756] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3684.219534] ? do_raw_spin_unlock+0x75/0x190 [ 3684.221061] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3684.223530] ? do_wait_intr+0xf0/0xf0 [ 3684.225407] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3684.227861] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3684.230612] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3684.233636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3684.236314] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3684.238743] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3684.241304] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3684.243907] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3684.246793] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3684.249800] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3684.252113] lmv_intent_open+0x338/0x1150 [lmv] [ 3684.254743] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3684.257459] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3684.260368] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3684.262480] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3684.264711] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3684.267422] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3684.269825] ? do_raw_spin_unlock+0x75/0x190 [ 3684.271835] ? _raw_spin_unlock+0x12/0x30 [ 3684.273784] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3684.276677] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3684.278909] ll_atomic_open+0x36a/0xd70 [lustre] [ 3684.283016] lookup_open+0x356/0xb40 [ 3684.284294] do_last+0x313/0xfc0 [ 3684.285401] ? nd_jump_root+0xe5/0x160 [ 3684.296592] ? path_init+0x437/0x520 [ 3684.299524] path_openat+0xf7/0x500 [ 3684.300695] ? do_raw_spin_unlock+0x75/0x190 [ 3684.318019] do_filp_open+0x99/0x140 [ 3684.319252] ? getname_flags+0x6e/0x330 [ 3684.325533] ? __check_object_size+0xff/0x256 [ 3684.327589] ? do_raw_spin_unlock+0x75/0x190 [ 3684.336501] ? _raw_spin_unlock+0x12/0x30 [ 3684.337903] do_sys_openat2+0x2b4/0x410 [ 3684.339208] do_sys_open+0x73/0xa0 [ 3684.348312] __x64_sys_openat+0x24/0x30 [ 3684.350334] do_syscall_64+0xc1/0x440 [ 3684.359255] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3684.360962] RIP: 0033:0x7fdc0afbf332 [ 3684.368694] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3684.386243] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3684.392472] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3684.397605] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3684.403413] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3684.413136] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3684.419889] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3696.051020] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 3696.055365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3696.076956] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3696.081694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3696.087792] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3696.092991] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3696.097368] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3696.107643] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3696.117278] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3696.122347] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3696.128017] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3696.138214] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3696.148577] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3696.150967] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3696.162178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3696.165483] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3696.169318] Call Trace: [ 3696.173111] [ 3696.175930] ? show_regs.cold.9+0x22/0x2f [ 3696.177596] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3696.180175] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3696.184893] ? watchdog+0x40/0x40 [ 3696.186839] ? hrtimer_interrupt+0x128/0x2c0 [ 3696.192408] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3696.198461] ? apic_timer_interrupt+0xf/0x20 [ 3696.200700] [ 3696.201631] ? native_safe_halt+0x2e/0x50 [ 3696.204053] kvm_wait+0xcd/0xe0 [ 3696.205672] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3696.208689] do_raw_spin_lock+0x177/0x190 [ 3696.210898] _raw_spin_lock+0x21/0x30 [ 3696.213892] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3696.217712] process_one_work+0x2c8/0x700 [ 3696.219305] worker_thread+0x296/0x6e0 [ 3696.224745] ? rescuer_thread+0x570/0x570 [ 3696.225828] kthread+0x1d1/0x200 [ 3696.227166] ? set_kthread_struct+0x70/0x70 [ 3696.228528] ret_from_fork+0x1f/0x30 [ 3699.382192] rcu: INFO: rcu_sched self-detected stall on CPU [ 3699.390930] rcu: 0-...!: (970355 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 3699.408985] (t=1680039 jiffies g=270761 q=3102) [ 3699.416718] rcu: rcu_sched kthread starved for 1674971 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 3699.424227] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3699.434276] rcu: RCU grace-period kthread stack dump: [ 3699.438260] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3699.443879] Call Trace: [ 3699.445115] __schedule+0x351/0xcb0 [ 3699.447866] schedule+0xc0/0x180 [ 3699.449785] schedule_timeout+0xb4/0x190 [ 3699.455219] ? do_raw_spin_unlock+0x75/0x190 [ 3699.459366] ? __next_timer_interrupt+0x160/0x160 [ 3699.467024] ? prepare_to_swait_event+0xf3/0x140 [ 3699.473021] rcu_gp_kthread+0x8dd/0x15b0 [ 3699.477331] ? force_qs_rnp+0x2c0/0x2c0 [ 3699.485335] kthread+0x1d1/0x200 [ 3699.486584] ? set_kthread_struct+0x70/0x70 [ 3699.491932] ret_from_fork+0x1f/0x30 [ 3699.493971] rcu: Stack dump where RCU GP kthread last ran: [ 3699.501781] NMI backtrace for cpu 0 [ 3699.505266] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3699.519014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3699.537801] Call Trace: [ 3699.542042] [ 3699.546793] ? dump_stack+0xbb/0x10e [ 3699.553648] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3699.555548] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3699.564795] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3699.569753] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3699.573033] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3699.575432] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 3699.578313] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 3699.585293] ? update_process_times+0x6d/0xc0 [ 3699.587998] ? tick_sched_handle.isra.18+0x27/0x90 [ 3699.592376] ? tick_sched_timer+0x53/0xb0 [ 3699.598352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3699.601597] ? tick_sched_handle.isra.18+0x90/0x90 [ 3699.604645] ? hrtimer_interrupt+0x128/0x2c0 [ 3699.606883] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3699.610317] ? apic_timer_interrupt+0xf/0x20 [ 3699.616636] [ 3699.617457] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3699.623254] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3699.636776] ? dump_stack+0xf6/0x10e [ 3699.638091] ? do_raw_spin_lock+0x177/0x190 [ 3699.639530] ? _raw_spin_lock+0x21/0x30 [ 3699.641641] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3699.644486] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3699.647108] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3699.648737] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3699.651796] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3699.654816] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3699.657469] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3699.659477] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3699.661546] ? do_wait_intr+0xf0/0xf0 [ 3699.663163] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3699.665732] ? kthread+0x1d1/0x200 [ 3699.667188] ? set_kthread_struct+0x70/0x70 [ 3699.668554] ? ret_from_fork+0x1f/0x30 [ 3699.670448] NMI backtrace for cpu 0 [ 3699.671916] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3699.676085] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3699.679540] Call Trace: [ 3699.680434] [ 3699.681129] ? dump_stack+0xbb/0x10e [ 3699.682262] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3699.684097] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3699.685552] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3699.687191] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3699.689151] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3699.691183] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3699.693388] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3699.696207] ? update_process_times+0x6d/0xc0 [ 3699.697643] ? tick_sched_handle.isra.18+0x27/0x90 [ 3699.699514] ? tick_sched_timer+0x53/0xb0 [ 3699.701521] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3699.703214] ? tick_sched_handle.isra.18+0x90/0x90 [ 3699.704972] ? hrtimer_interrupt+0x128/0x2c0 [ 3699.707133] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3699.709141] ? apic_timer_interrupt+0xf/0x20 [ 3699.711290] [ 3699.712290] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3699.714719] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3699.717568] ? dump_stack+0xf6/0x10e [ 3699.719557] ? do_raw_spin_lock+0x177/0x190 [ 3699.722239] ? _raw_spin_lock+0x21/0x30 [ 3699.724457] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3699.732201] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3699.736951] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3699.740244] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3699.742446] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3699.745322] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3699.748352] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3699.751058] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3699.753487] ? do_wait_intr+0xf0/0xf0 [ 3699.755322] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3699.757584] ? kthread+0x1d1/0x200 [ 3699.759293] ? set_kthread_struct+0x70/0x70 [ 3699.764722] ? ret_from_fork+0x1f/0x30 [ 3700.054255] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3700.063228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3700.116212] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3700.125531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3700.134901] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3700.140691] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3700.169736] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3700.180368] RAX: 0000000000007d84 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3700.192356] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3700.197759] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3700.206594] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3700.216919] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3700.223597] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3700.235928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3700.237904] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3700.242511] Call Trace: [ 3700.245905] [ 3700.247043] ? show_regs.cold.9+0x22/0x2f [ 3700.253562] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3700.261042] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3700.266798] ? watchdog+0x40/0x40 [ 3700.267812] ? hrtimer_interrupt+0x128/0x2c0 [ 3700.278801] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3700.282230] ? apic_timer_interrupt+0xf/0x20 [ 3700.285303] [ 3700.286222] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3700.297193] do_raw_spin_lock+0x177/0x190 [ 3700.301037] _raw_spin_lock+0x21/0x30 [ 3700.305423] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3700.320612] ? do_raw_spin_unlock+0x75/0x190 [ 3700.327388] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3700.333558] ? do_wait_intr+0xf0/0xf0 [ 3700.345598] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3700.349983] mdc_close+0x53b/0xd70 [mdc] [ 3700.356411] lmv_close+0x18d/0x490 [lmv] [ 3700.361454] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3700.374796] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3700.379480] ll_md_close+0x274/0x800 [lustre] [ 3700.387882] ll_file_release+0x128/0xc50 [lustre] [ 3700.392445] __fput+0x10d/0x440 [ 3700.393674] ____fput+0x12/0x20 [ 3700.400321] task_work_run+0xc6/0x110 [ 3700.402628] exit_to_usermode_loop+0x1dd/0x1f0 [ 3700.411070] do_syscall_64+0x3d6/0x440 [ 3700.416343] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3700.422554] RIP: 0033:0x7fdc0afbfcd8 [ 3700.425378] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3700.435128] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3700.439485] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3700.444524] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3700.447476] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3700.450849] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3700.456296] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3707.360449] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1682s! [ 3707.366374] Showing busy workqueues and worker pools: [ 3707.373413] workqueue events: flags=0x0 [ 3707.379258] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3707.392654] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3707.400062] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3707.406435] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3707.417360] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3707.431213] pending: kfree_rcu_monitor [ 3707.435438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3707.447508] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 3707.469593] workqueue events_long: flags=0x0 [ 3707.474196] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3707.486551] pending: lu_objects_flush [obdclass] [ 3707.490463] workqueue events_unbound: flags=0x2 [ 3707.494621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3707.496906] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3707.509826] workqueue events_power_efficient: flags=0x80 [ 3707.515511] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3707.526730] pending: neigh_periodic_work, check_lifetime [ 3707.536438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3707.548183] pending: do_cache_clean, neigh_periodic_work [ 3707.562166] workqueue mm_percpu_wq: flags=0x8 [ 3707.568129] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3707.578891] pending: lru_add_drain_per_cpu [ 3707.584646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3707.595243] pending: vmstat_update, lru_add_drain_per_cpu [ 3707.603445] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3707.609966] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3707.622318] workqueue writeback: flags=0x4a [ 3707.625548] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3707.632215] pending: wb_workfn, wb_workfn [ 3707.637161] workqueue xprtiod: flags=0xa [ 3707.643574] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3707.656157] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3707.664853] workqueue ipv6_addrconf: flags=0x40008 [ 3707.677421] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3707.689389] pending: addrconf_verify_work [ 3707.698462] workqueue knbd0-recv: flags=0x1a [ 3707.706287] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3707.716955] in-flight: 655:recv_work [ 3707.723874] workqueue ptlrpc_pinger: flags=0x2 [ 3707.729349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3707.737713] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3707.750978] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1682s workers=4 idle: 36396 38 11188 [ 3707.757920] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3698s workers=2 idle: 658 [ 3712.053070] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 3712.057440] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3712.082102] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3712.085791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3712.088676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3712.094790] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3712.110581] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3712.114606] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3712.120621] RDX: 0000000000005fa1 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3712.125283] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3712.129350] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3712.133603] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3712.136282] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3712.141316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3712.154006] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3712.156505] Call Trace: [ 3712.161338] [ 3712.162997] ? show_regs.cold.9+0x22/0x2f [ 3712.166235] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3712.169459] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3712.172501] ? watchdog+0x40/0x40 [ 3712.174395] ? hrtimer_interrupt+0x128/0x2c0 [ 3712.177326] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3712.180420] ? apic_timer_interrupt+0xf/0x20 [ 3712.182696] [ 3712.184293] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3712.187622] do_raw_spin_lock+0x177/0x190 [ 3712.190486] _raw_spin_lock+0x21/0x30 [ 3712.192339] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3712.196518] ? do_raw_spin_unlock+0x75/0x190 [ 3712.199114] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3712.201936] ? do_wait_intr+0xf0/0xf0 [ 3712.204393] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3712.207903] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3712.211516] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3712.216010] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3712.219325] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3712.222089] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3712.225111] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3712.227150] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3712.230907] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3712.234733] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3712.237982] lmv_intent_open+0x338/0x1150 [lmv] [ 3712.242280] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3712.246215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3712.252721] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3712.254224] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3712.257415] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3712.261096] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3712.267270] ? do_raw_spin_unlock+0x75/0x190 [ 3712.268374] ? _raw_spin_unlock+0x12/0x30 [ 3712.272032] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3712.278599] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3712.282575] ll_atomic_open+0x36a/0xd70 [lustre] [ 3712.285247] lookup_open+0x356/0xb40 [ 3712.286664] do_last+0x313/0xfc0 [ 3712.289963] ? nd_jump_root+0xe5/0x160 [ 3712.291361] ? path_init+0x437/0x520 [ 3712.293308] path_openat+0xf7/0x500 [ 3712.295369] ? do_raw_spin_unlock+0x75/0x190 [ 3712.298328] do_filp_open+0x99/0x140 [ 3712.300353] ? getname_flags+0x6e/0x330 [ 3712.304144] ? __check_object_size+0xff/0x256 [ 3712.306274] ? do_raw_spin_unlock+0x75/0x190 [ 3712.309249] ? _raw_spin_unlock+0x12/0x30 [ 3712.312226] do_sys_openat2+0x2b4/0x410 [ 3712.315380] do_sys_open+0x73/0xa0 [ 3712.318135] __x64_sys_openat+0x24/0x30 [ 3712.320726] do_syscall_64+0xc1/0x440 [ 3712.322940] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3712.327120] RIP: 0033:0x7fdc0afbf332 [ 3712.330672] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3712.345745] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3712.352623] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3712.357515] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3712.360701] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3712.365146] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3712.375274] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3724.052143] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 3724.056122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3724.091808] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3724.096667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3724.100315] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3724.107612] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3724.111179] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3724.127479] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3724.131437] RAX: 0000000000007f58 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3724.136305] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3724.139585] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3724.141341] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3724.146077] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3724.150003] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3724.153103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3724.155052] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3724.158373] Call Trace: [ 3724.159076] [ 3724.159759] ? show_regs.cold.9+0x22/0x2f [ 3724.161755] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3724.163176] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3724.164597] ? watchdog+0x40/0x40 [ 3724.165794] ? hrtimer_interrupt+0x128/0x2c0 [ 3724.167198] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3724.168708] ? apic_timer_interrupt+0xf/0x20 [ 3724.170127] [ 3724.170817] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3724.172574] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3724.174297] do_raw_spin_lock+0x177/0x190 [ 3724.175562] _raw_spin_lock+0x21/0x30 [ 3724.176614] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3724.178577] process_one_work+0x2c8/0x700 [ 3724.179811] worker_thread+0x296/0x6e0 [ 3724.181015] ? rescuer_thread+0x570/0x570 [ 3724.182060] kthread+0x1d1/0x200 [ 3724.183160] ? set_kthread_struct+0x70/0x70 [ 3724.184655] ret_from_fork+0x1f/0x30 [ 3728.057804] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 3728.071162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3728.153012] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3728.172369] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3728.183340] RIP: 0010:_raw_spin_unlock_irqrestore+0x4f/0x90 [ 3728.187352] Code: 01 48 83 05 e2 91 f9 01 01 9c 58 0f 1f 44 00 00 48 83 05 db 91 f9 01 01 48 83 05 3b 93 f9 01 01 f6 c4 02 75 2b 48 89 df 57 9d <0f> 1f 44 00 00 48 83 05 3c 93 f9 01 01 48 83 05 3c 93 f9 01 01 65 [ 3728.207588] RSP: 0018:ffffba8e40d8ced8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3728.212941] RAX: 0000000000000006 RBX: 0000000000000212 RCX: ffff9fe618df2840 [ 3728.220239] RDX: 0000000000000018 RSI: 0000000000000212 RDI: 0000000000000212 [ 3728.227245] RBP: ffff9fe6421b3d38 R08: 0000000000000000 R09: 0000000000000000 [ 3728.231031] R10: 00000000000f4240 R11: 000000000000ba3d R12: ffff9fe6421b3440 [ 3728.239657] R13: 0000000000000000 R14: 0000000000000000 R15: ffff9fe6421b3380 [ 3728.243860] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3728.261922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3728.269960] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3728.280144] Call Trace: [ 3728.281163] [ 3728.282309] ? show_regs.cold.9+0x22/0x2f [ 3728.285223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3728.288657] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3728.293275] ? watchdog+0x40/0x40 [ 3728.299214] ? hrtimer_interrupt+0x128/0x2c0 [ 3728.308953] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3728.314615] ? apic_timer_interrupt+0xf/0x20 [ 3728.323971] ? apic_timer_interrupt+0xa/0x20 [ 3728.330008] ? _raw_spin_unlock_irqrestore+0x4f/0x90 [ 3728.336436] update_blocked_averages+0x763/0x830 [ 3728.338113] ? kvm_clock_get_cycles+0x2c/0x50 [ 3728.350525] run_rebalance_domains+0x4f/0xa0 [ 3728.356126] __do_softirq+0x134/0x4e8 [ 3728.359953] irq_exit_rcu+0x1c6/0x1e0 [ 3728.361199] irq_exit+0x12/0x30 [ 3728.371875] smp_apic_timer_interrupt+0xe0/0x270 [ 3728.373525] apic_timer_interrupt+0xf/0x20 [ 3728.377324] [ 3728.378389] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3728.386862] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3728.402093] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3728.408638] RAX: 0000000000004185 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3728.412369] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3728.420244] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3728.424537] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3728.433650] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3728.438933] do_raw_spin_lock+0x177/0x190 [ 3728.444388] _raw_spin_lock+0x21/0x30 [ 3728.446911] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3728.454395] ? do_raw_spin_unlock+0x75/0x190 [ 3728.464048] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3728.466052] ? do_wait_intr+0xf0/0xf0 [ 3728.469240] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3728.473383] mdc_close+0x53b/0xd70 [mdc] [ 3728.475693] lmv_close+0x18d/0x490 [lmv] [ 3728.477913] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3728.480961] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3728.484376] ll_md_close+0x274/0x800 [lustre] [ 3728.487708] ll_file_release+0x128/0xc50 [lustre] [ 3728.492060] __fput+0x10d/0x440 [ 3728.506698] ____fput+0x12/0x20 [ 3728.515095] task_work_run+0xc6/0x110 [ 3728.522098] exit_to_usermode_loop+0x1dd/0x1f0 [ 3728.530741] do_syscall_64+0x3d6/0x440 [ 3728.536062] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3728.542971] RIP: 0033:0x7fdc0afbfcd8 [ 3728.545990] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3728.566035] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3728.577916] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3728.585238] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3728.592413] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3728.601794] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3728.612943] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3732.048010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3732.058056] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3732.101582] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3732.106868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3732.113880] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3732.116772] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3732.131459] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3732.136337] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3732.139395] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3732.143147] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3732.148801] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3732.153305] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3732.160118] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3732.170052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3732.176962] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3732.182436] Call Trace: [ 3732.183587] [ 3732.184540] ? show_regs.cold.9+0x22/0x2f [ 3732.186383] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3732.189192] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3732.191649] ? watchdog+0x40/0x40 [ 3732.193381] ? hrtimer_interrupt+0x128/0x2c0 [ 3732.202914] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3732.206454] ? apic_timer_interrupt+0xf/0x20 [ 3732.211008] [ 3732.212884] ? native_safe_halt+0x2e/0x50 [ 3732.221904] kvm_wait+0xcd/0xe0 [ 3732.223912] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3732.226323] ? dump_stack+0xf6/0x10e [ 3732.231723] do_raw_spin_lock+0x177/0x190 [ 3732.233323] _raw_spin_lock+0x21/0x30 [ 3732.237310] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3732.242053] obd_mod_free+0x46/0x140 [ptlrpc] [ 3732.245289] mdc_commit_open+0x91/0xb0 [mdc] [ 3732.247490] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3732.249943] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3732.256166] after_reply+0xe2c/0x1780 [ptlrpc] [ 3732.262475] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3732.267769] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3732.284770] ? do_wait_intr+0xf0/0xf0 [ 3732.287973] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3732.293045] kthread+0x1d1/0x200 [ 3732.294272] ? set_kthread_struct+0x70/0x70 [ 3732.296895] ret_from_fork+0x1f/0x30 [ 3738.079046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1713s! [ 3738.081927] Showing busy workqueues and worker pools: [ 3738.085829] workqueue events: flags=0x0 [ 3738.088518] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3738.095478] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3738.102343] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3738.107807] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3738.112180] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3738.116484] pending: kfree_rcu_monitor [ 3738.119239] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3738.123377] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 3738.134613] workqueue events_long: flags=0x0 [ 3738.137325] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3738.145619] pending: lu_objects_flush [obdclass] [ 3738.149510] workqueue events_unbound: flags=0x2 [ 3738.154515] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3738.163690] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3738.180152] workqueue events_power_efficient: flags=0x80 [ 3738.181918] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3738.190451] pending: neigh_periodic_work, check_lifetime [ 3738.196354] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3738.205846] pending: do_cache_clean, neigh_periodic_work [ 3738.211734] workqueue mm_percpu_wq: flags=0x8 [ 3738.213679] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3738.220606] pending: lru_add_drain_per_cpu [ 3738.222040] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3738.229904] pending: vmstat_update, lru_add_drain_per_cpu [ 3738.236279] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3738.245872] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3738.253865] workqueue writeback: flags=0x4a [ 3738.260417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3738.263646] pending: wb_workfn, wb_workfn [ 3738.265103] workqueue xprtiod: flags=0xa [ 3738.268263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3738.272712] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3738.280548] workqueue ipv6_addrconf: flags=0x40008 [ 3738.286373] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3738.291878] pending: addrconf_verify_work [ 3738.296732] workqueue knbd0-recv: flags=0x1a [ 3738.302948] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3738.312243] in-flight: 655:recv_work [ 3738.319438] workqueue ptlrpc_pinger: flags=0x2 [ 3738.325611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3738.331989] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3738.338107] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1713s workers=4 idle: 36396 38 11188 [ 3738.349896] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3728s workers=2 idle: 658 [ 3740.052035] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 3740.055775] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3740.092649] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3740.101029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3740.104745] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3740.108325] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3740.118897] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3740.122752] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3740.126865] RDX: 0000000000005540 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3740.131393] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3740.135340] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3740.138382] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3740.143309] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3740.147667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3740.150722] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3740.155328] Call Trace: [ 3740.157138] [ 3740.158732] ? show_regs.cold.9+0x22/0x2f [ 3740.160892] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3740.163046] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3740.165519] ? watchdog+0x40/0x40 [ 3740.167027] ? hrtimer_interrupt+0x128/0x2c0 [ 3740.169177] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3740.172554] ? apic_timer_interrupt+0xf/0x20 [ 3740.175710] [ 3740.177099] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3740.181215] do_raw_spin_lock+0x177/0x190 [ 3740.183060] _raw_spin_lock+0x21/0x30 [ 3740.185739] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3740.189700] ? do_raw_spin_unlock+0x75/0x190 [ 3740.193253] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3740.197464] ? do_wait_intr+0xf0/0xf0 [ 3740.199647] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3740.202960] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3740.206580] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3740.210714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3740.213944] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3740.217428] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3740.220891] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3740.223341] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3740.226501] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3740.229988] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3740.232025] lmv_intent_open+0x338/0x1150 [lmv] [ 3740.234270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3740.236265] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3740.239335] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3740.242140] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3740.245310] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3740.247870] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3740.250743] ? do_raw_spin_unlock+0x75/0x190 [ 3740.252930] ? _raw_spin_unlock+0x12/0x30 [ 3740.254720] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3740.257375] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3740.259574] ll_atomic_open+0x36a/0xd70 [lustre] [ 3740.262119] lookup_open+0x356/0xb40 [ 3740.263741] do_last+0x313/0xfc0 [ 3740.265374] ? nd_jump_root+0xe5/0x160 [ 3740.267082] ? path_init+0x437/0x520 [ 3740.268682] path_openat+0xf7/0x500 [ 3740.270527] ? do_raw_spin_unlock+0x75/0x190 [ 3740.274394] do_filp_open+0x99/0x140 [ 3740.277743] ? getname_flags+0x6e/0x330 [ 3740.279805] ? __check_object_size+0xff/0x256 [ 3740.286394] ? do_raw_spin_unlock+0x75/0x190 [ 3740.289429] ? _raw_spin_unlock+0x12/0x30 [ 3740.292122] do_sys_openat2+0x2b4/0x410 [ 3740.293814] do_sys_open+0x73/0xa0 [ 3740.295520] __x64_sys_openat+0x24/0x30 [ 3740.297549] do_syscall_64+0xc1/0x440 [ 3740.303313] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3740.305557] RIP: 0033:0x7fdc0afbf332 [ 3740.310933] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3740.328502] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3740.334862] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3740.339857] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3740.346462] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3740.353362] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3740.359148] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3752.051109] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3752.057500] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3752.080050] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3752.085317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3752.089489] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3752.092578] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3752.095958] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3752.105687] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3752.109894] RAX: 000000000000714e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3752.112676] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3752.115940] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3752.119520] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3752.123398] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3752.127084] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3752.130956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3752.133322] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3752.136127] Call Trace: [ 3752.137132] [ 3752.137879] ? show_regs.cold.9+0x22/0x2f [ 3752.144101] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3752.146558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3752.148990] ? watchdog+0x40/0x40 [ 3752.150724] ? hrtimer_interrupt+0x128/0x2c0 [ 3752.152810] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3752.155494] ? apic_timer_interrupt+0xf/0x20 [ 3752.157604] [ 3752.158658] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3752.161013] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3752.167703] do_raw_spin_lock+0x177/0x190 [ 3752.170656] _raw_spin_lock+0x21/0x30 [ 3752.172002] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3752.175794] process_one_work+0x2c8/0x700 [ 3752.179587] worker_thread+0x296/0x6e0 [ 3752.182285] ? rescuer_thread+0x570/0x570 [ 3752.185539] kthread+0x1d1/0x200 [ 3752.186939] ? set_kthread_struct+0x70/0x70 [ 3752.190806] ret_from_fork+0x1f/0x30 [ 3756.054050] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 3756.057434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3756.077540] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3756.081928] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3756.085273] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3756.087869] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3756.095227] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3756.097795] RAX: 0000000000006587 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3756.101662] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3756.107882] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3756.111869] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3756.117559] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3756.122846] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3756.129154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3756.133603] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3756.139365] Call Trace: [ 3756.141812] [ 3756.142944] ? show_regs.cold.9+0x22/0x2f [ 3756.146172] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3756.150502] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3756.153212] ? watchdog+0x40/0x40 [ 3756.156949] ? hrtimer_interrupt+0x128/0x2c0 [ 3756.160225] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3756.164640] ? apic_timer_interrupt+0xf/0x20 [ 3756.168188] [ 3756.169283] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3756.174242] do_raw_spin_lock+0x177/0x190 [ 3756.178534] _raw_spin_lock+0x21/0x30 [ 3756.181599] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3756.184052] ? do_raw_spin_unlock+0x75/0x190 [ 3756.190063] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3756.196619] ? do_wait_intr+0xf0/0xf0 [ 3756.200334] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3756.206630] mdc_close+0x53b/0xd70 [mdc] [ 3756.213560] lmv_close+0x18d/0x490 [lmv] [ 3756.217554] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3756.224528] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3756.228361] ll_md_close+0x274/0x800 [lustre] [ 3756.231371] ll_file_release+0x128/0xc50 [lustre] [ 3756.234268] __fput+0x10d/0x440 [ 3756.240073] ____fput+0x12/0x20 [ 3756.241426] task_work_run+0xc6/0x110 [ 3756.243376] exit_to_usermode_loop+0x1dd/0x1f0 [ 3756.246282] do_syscall_64+0x3d6/0x440 [ 3756.247916] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3756.250424] RIP: 0033:0x7fdc0afbfcd8 [ 3756.251903] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3756.262568] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3756.272275] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3756.276289] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3756.279786] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3756.284245] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3756.289244] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3764.051844] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3764.056068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3764.091290] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3764.097915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3764.107836] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3764.114810] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3764.129271] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3764.135148] RAX: 0000000000004c3b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3764.144117] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3764.150844] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3764.158289] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3764.167340] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3764.178777] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3764.185096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3764.189040] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3764.195263] Call Trace: [ 3764.197146] [ 3764.199451] ? show_regs.cold.9+0x22/0x2f [ 3764.202216] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3764.206725] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3764.210823] ? watchdog+0x40/0x40 [ 3764.212253] ? hrtimer_interrupt+0x128/0x2c0 [ 3764.218522] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3764.222499] ? apic_timer_interrupt+0xf/0x20 [ 3764.225079] [ 3764.226002] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3764.229659] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3764.234086] ? dump_stack+0xf6/0x10e [ 3764.235460] do_raw_spin_lock+0x177/0x190 [ 3764.238050] _raw_spin_lock+0x21/0x30 [ 3764.242110] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3764.247269] obd_mod_free+0x46/0x140 [ptlrpc] [ 3764.250472] mdc_commit_open+0x91/0xb0 [mdc] [ 3764.254878] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3764.261971] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3764.265508] after_reply+0xe2c/0x1780 [ptlrpc] [ 3764.270289] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3764.276818] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3764.287082] ? do_wait_intr+0xf0/0xf0 [ 3764.289504] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3764.302396] kthread+0x1d1/0x200 [ 3764.315254] ? set_kthread_struct+0x70/0x70 [ 3764.317995] ret_from_fork+0x1f/0x30 [ 3768.052011] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 3768.059067] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3768.096608] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3768.115118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3768.122499] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3768.125358] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3768.146307] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3768.151833] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3768.156313] RDX: 0000000000000ebf RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3768.163055] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3768.172540] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3768.185380] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3768.195879] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3768.202526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3768.204502] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3768.214928] Call Trace: [ 3768.222533] [ 3768.223367] ? show_regs.cold.9+0x22/0x2f [ 3768.224862] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3768.230879] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3768.233904] ? watchdog+0x40/0x40 [ 3768.236767] ? hrtimer_interrupt+0x128/0x2c0 [ 3768.244859] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3768.252925] ? apic_timer_interrupt+0xf/0x20 [ 3768.260593] [ 3768.261359] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3768.269760] do_raw_spin_lock+0x177/0x190 [ 3768.272362] _raw_spin_lock+0x21/0x30 [ 3768.274667] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3768.288000] ? do_raw_spin_unlock+0x75/0x190 [ 3768.292674] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3768.301267] ? do_wait_intr+0xf0/0xf0 [ 3768.305977] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3768.307952] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3768.311373] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3768.319224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3768.321215] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3768.340494] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3768.345732] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3768.351653] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3768.357684] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3768.367470] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3768.374444] lmv_intent_open+0x338/0x1150 [lmv] [ 3768.378750] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3768.389145] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3768.395875] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3768.397455] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3768.405526] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3768.412360] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3768.419116] ? do_raw_spin_unlock+0x75/0x190 [ 3768.428291] ? _raw_spin_unlock+0x12/0x30 [ 3768.433634] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3768.438712] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3768.451772] ll_atomic_open+0x36a/0xd70 [lustre] [ 3768.463739] lookup_open+0x356/0xb40 [ 3768.471822] do_last+0x313/0xfc0 [ 3768.475132] ? nd_jump_root+0xe5/0x160 [ 3768.477597] ? path_init+0x437/0x520 [ 3768.484285] path_openat+0xf7/0x500 [ 3768.487629] ? do_raw_spin_unlock+0x75/0x190 [ 3768.492388] do_filp_open+0x99/0x140 [ 3768.493598] ? getname_flags+0x6e/0x330 [ 3768.503162] ? __check_object_size+0xff/0x256 [ 3768.508132] ? do_raw_spin_unlock+0x75/0x190 [ 3768.513824] ? _raw_spin_unlock+0x12/0x30 [ 3768.520028] do_sys_openat2+0x2b4/0x410 [ 3768.526442] do_sys_open+0x73/0xa0 [ 3768.527711] __x64_sys_openat+0x24/0x30 [ 3768.530225] do_syscall_64+0xc1/0x440 [ 3768.532714] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3768.536668] RIP: 0033:0x7fdc0afbf332 [ 3768.549568] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3768.574695] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3768.582812] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3768.589388] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3768.604236] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3768.610158] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3768.624622] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3768.799034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1743s! [ 3768.808229] Showing busy workqueues and worker pools: [ 3768.810353] workqueue events: flags=0x0 [ 3768.812041] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3768.814675] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3768.817652] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3768.819478] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3768.824295] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3768.826954] pending: kfree_rcu_monitor [ 3768.828912] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3768.831906] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 3768.836884] workqueue events_long: flags=0x0 [ 3768.838441] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3768.841301] pending: lu_objects_flush [obdclass] [ 3768.843563] workqueue events_unbound: flags=0x2 [ 3768.845594] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3768.848351] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3768.853021] workqueue events_power_efficient: flags=0x80 [ 3768.855528] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3768.858218] pending: neigh_periodic_work, check_lifetime [ 3768.860807] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3768.863525] pending: do_cache_clean, neigh_periodic_work [ 3768.866202] workqueue mm_percpu_wq: flags=0x8 [ 3768.868286] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3768.871241] pending: lru_add_drain_per_cpu [ 3768.873361] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3768.876148] pending: vmstat_update, lru_add_drain_per_cpu [ 3768.878316] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3768.881184] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3768.883913] workqueue writeback: flags=0x4a [ 3768.886111] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3768.889207] pending: wb_workfn, wb_workfn [ 3768.891186] workqueue xprtiod: flags=0xa [ 3768.892826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3768.895357] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3768.899375] workqueue ipv6_addrconf: flags=0x40008 [ 3768.901265] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3768.904552] pending: addrconf_verify_work [ 3768.906128] workqueue knbd0-recv: flags=0x1a [ 3768.908056] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3768.910767] in-flight: 655:recv_work [ 3768.912352] workqueue ptlrpc_pinger: flags=0x2 [ 3768.914298] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3768.916713] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3768.919429] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1743s workers=4 idle: 36396 38 11188 [ 3768.923099] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3759s workers=2 idle: 658 [ 3780.051180] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3780.060572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3780.092034] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3780.101805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3780.108322] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3780.113679] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3780.118522] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3780.131997] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3780.136911] RAX: 0000000000007f8e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3780.145336] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3780.153056] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3780.162435] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3780.178754] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3780.183696] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3780.193584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3780.201438] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3780.216199] Call Trace: [ 3780.226728] [ 3780.227512] ? show_regs.cold.9+0x22/0x2f [ 3780.228860] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3780.235691] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3780.243295] ? watchdog+0x40/0x40 [ 3780.248944] ? hrtimer_interrupt+0x128/0x2c0 [ 3780.255542] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3780.257219] ? apic_timer_interrupt+0xf/0x20 [ 3780.264832] [ 3780.269586] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3780.275019] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3780.277525] do_raw_spin_lock+0x177/0x190 [ 3780.279109] _raw_spin_lock+0x21/0x30 [ 3780.280921] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3780.283590] process_one_work+0x2c8/0x700 [ 3780.285407] worker_thread+0x296/0x6e0 [ 3780.286923] ? rescuer_thread+0x570/0x570 [ 3780.288670] kthread+0x1d1/0x200 [ 3780.290039] ? set_kthread_struct+0x70/0x70 [ 3780.292152] ret_from_fork+0x1f/0x30 [ 3784.054031] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 3784.058063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3784.110307] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3784.117124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3784.125389] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3784.131423] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3784.149729] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3784.159548] RAX: 0000000000003785 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3784.165379] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3784.170547] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3784.181292] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3784.186046] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3784.190609] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3784.198575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3784.203217] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3784.209305] Call Trace: [ 3784.211760] [ 3784.212424] ? show_regs.cold.9+0x22/0x2f [ 3784.214659] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3784.216193] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3784.222546] ? watchdog+0x40/0x40 [ 3784.225546] ? hrtimer_interrupt+0x128/0x2c0 [ 3784.228484] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3784.233403] ? apic_timer_interrupt+0xf/0x20 [ 3784.238170] [ 3784.241009] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3784.244675] do_raw_spin_lock+0x177/0x190 [ 3784.253192] _raw_spin_lock+0x21/0x30 [ 3784.257739] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3784.261120] ? do_raw_spin_unlock+0x75/0x190 [ 3784.265351] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3784.272262] ? do_wait_intr+0xf0/0xf0 [ 3784.273430] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3784.276397] mdc_close+0x53b/0xd70 [mdc] [ 3784.278477] lmv_close+0x18d/0x490 [lmv] [ 3784.283765] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3784.285438] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3784.292455] ll_md_close+0x274/0x800 [lustre] [ 3784.295276] ll_file_release+0x128/0xc50 [lustre] [ 3784.299922] __fput+0x10d/0x440 [ 3784.302057] ____fput+0x12/0x20 [ 3784.303634] task_work_run+0xc6/0x110 [ 3784.305954] exit_to_usermode_loop+0x1dd/0x1f0 [ 3784.310737] do_syscall_64+0x3d6/0x440 [ 3784.312266] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3784.318407] RIP: 0033:0x7fdc0afbfcd8 [ 3784.319598] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3784.329546] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3784.332975] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3784.336035] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3784.339056] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3784.342416] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3784.345277] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3796.053400] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 3796.058695] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 3796.059899] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3796.060039] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3796.060046] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3796.060048] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3796.060064] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3796.060068] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3796.060073] RAX: 0000000000005a8b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3796.060076] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3796.060078] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3796.060080] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3796.060135] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3796.060139] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3796.060142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3796.060144] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3796.060153] Call Trace: [ 3796.060158] [ 3796.060164] ? show_regs.cold.9+0x22/0x2f [ 3796.060174] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3796.060182] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3796.060186] ? watchdog+0x40/0x40 [ 3796.060190] ? hrtimer_interrupt+0x128/0x2c0 [ 3796.060194] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3796.060202] ? apic_timer_interrupt+0xf/0x20 [ 3796.060207] [ 3796.060210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3796.060215] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3796.060219] ? dump_stack+0xf6/0x10e [ 3796.060224] do_raw_spin_lock+0x177/0x190 [ 3796.060229] _raw_spin_lock+0x21/0x30 [ 3796.060235] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3796.060657] obd_mod_free+0x46/0x140 [ptlrpc] [ 3796.060923] mdc_commit_open+0x91/0xb0 [mdc] [ 3796.060961] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3796.061289] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3796.061508] after_reply+0xe2c/0x1780 [ptlrpc] [ 3796.062610] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3796.062889] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3796.063162] ? do_wait_intr+0xf0/0xf0 [ 3796.063173] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3796.063518] kthread+0x1d1/0x200 [ 3796.063525] ? set_kthread_struct+0x70/0x70 [ 3796.063529] ret_from_fork+0x1f/0x30 [ 3796.345463] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3796.395557] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3796.399709] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3796.405729] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3796.409292] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3796.418529] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3796.421645] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3796.423833] RDX: 000000000000747e RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3796.429368] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3796.435787] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3796.438474] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3796.441557] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3796.445659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3796.448179] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3796.454192] Call Trace: [ 3796.454870] [ 3796.459519] ? show_regs.cold.9+0x22/0x2f [ 3796.464783] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3796.466443] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3796.474286] ? watchdog+0x40/0x40 [ 3796.475325] ? hrtimer_interrupt+0x128/0x2c0 [ 3796.476804] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3796.480957] ? apic_timer_interrupt+0xf/0x20 [ 3796.484801] [ 3796.485748] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3796.488273] do_raw_spin_lock+0x177/0x190 [ 3796.493874] _raw_spin_lock+0x21/0x30 [ 3796.499789] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3796.505526] ? do_raw_spin_unlock+0x75/0x190 [ 3796.507087] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3796.517952] ? do_wait_intr+0xf0/0xf0 [ 3796.520633] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3796.523224] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3796.525776] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3796.530417] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3796.533214] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3796.537594] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3796.538914] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3796.540280] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3796.548246] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3796.552710] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3796.554958] lmv_intent_open+0x338/0x1150 [lmv] [ 3796.557604] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3796.562922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3796.565284] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3796.567555] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3796.569495] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3796.571650] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3796.573596] ? do_raw_spin_unlock+0x75/0x190 [ 3796.575854] ? _raw_spin_unlock+0x12/0x30 [ 3796.578703] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3796.580578] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3796.583257] ll_atomic_open+0x36a/0xd70 [lustre] [ 3796.586019] lookup_open+0x356/0xb40 [ 3796.586892] do_last+0x313/0xfc0 [ 3796.588815] ? nd_jump_root+0xe5/0x160 [ 3796.591430] ? path_init+0x437/0x520 [ 3796.593635] path_openat+0xf7/0x500 [ 3796.595836] ? do_raw_spin_unlock+0x75/0x190 [ 3796.598572] do_filp_open+0x99/0x140 [ 3796.599885] ? getname_flags+0x6e/0x330 [ 3796.602046] ? __check_object_size+0xff/0x256 [ 3796.603182] ? do_raw_spin_unlock+0x75/0x190 [ 3796.604933] ? _raw_spin_unlock+0x12/0x30 [ 3796.606368] do_sys_openat2+0x2b4/0x410 [ 3796.609613] do_sys_open+0x73/0xa0 [ 3796.610894] __x64_sys_openat+0x24/0x30 [ 3796.612484] do_syscall_64+0xc1/0x440 [ 3796.615206] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3796.617401] RIP: 0033:0x7fdc0afbf332 [ 3796.620225] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3796.632609] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3796.644897] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3796.647018] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3796.657250] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3796.661550] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3796.665856] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3799.007272] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1774s! [ 3799.015841] Showing busy workqueues and worker pools: [ 3799.021145] workqueue events: flags=0x0 [ 3799.024217] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3799.029283] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3799.033839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3799.038688] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3799.047754] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3799.055245] pending: kfree_rcu_monitor [ 3799.057985] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3799.063006] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 3799.075483] workqueue events_long: flags=0x0 [ 3799.078768] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3799.083180] pending: lu_objects_flush [obdclass] [ 3799.089803] workqueue events_unbound: flags=0x2 [ 3799.091875] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3799.095799] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3799.102376] workqueue events_power_efficient: flags=0x80 [ 3799.105250] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3799.112229] pending: neigh_periodic_work, check_lifetime [ 3799.116024] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3799.120618] pending: do_cache_clean, neigh_periodic_work [ 3799.123803] workqueue mm_percpu_wq: flags=0x8 [ 3799.126456] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3799.132087] pending: lru_add_drain_per_cpu [ 3799.134302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3799.141522] pending: vmstat_update, lru_add_drain_per_cpu [ 3799.146096] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3799.148408] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3799.151014] workqueue writeback: flags=0x4a [ 3799.153603] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3799.156852] pending: wb_workfn, wb_workfn [ 3799.158800] workqueue xprtiod: flags=0xa [ 3799.161746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3799.164388] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3799.168878] workqueue ipv6_addrconf: flags=0x40008 [ 3799.171015] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3799.173125] pending: addrconf_verify_work [ 3799.175526] workqueue knbd0-recv: flags=0x1a [ 3799.177746] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3799.184131] in-flight: 655:recv_work [ 3799.185883] workqueue ptlrpc_pinger: flags=0x2 [ 3799.189524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3799.192390] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3799.195358] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1774s workers=4 idle: 36396 38 11188 [ 3799.199387] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3789s workers=2 idle: 658 [ 3808.054024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3808.056678] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3808.120443] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3808.129241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3808.142290] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3808.152721] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3808.162415] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3808.180934] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3808.187182] RAX: 0000000000007e5e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3808.190586] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3808.198158] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3808.207455] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3808.217933] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3808.226922] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3808.235340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3808.237382] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3808.252806] Call Trace: [ 3808.253659] [ 3808.268432] ? show_regs.cold.9+0x22/0x2f [ 3808.269987] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3808.272921] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3808.274676] ? watchdog+0x40/0x40 [ 3808.275919] ? hrtimer_interrupt+0x128/0x2c0 [ 3808.281598] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3808.286866] ? apic_timer_interrupt+0xf/0x20 [ 3808.289384] [ 3808.292585] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3808.296203] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3808.303735] do_raw_spin_lock+0x177/0x190 [ 3808.305146] _raw_spin_lock+0x21/0x30 [ 3808.307133] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3808.310697] process_one_work+0x2c8/0x700 [ 3808.321138] worker_thread+0x296/0x6e0 [ 3808.322937] ? rescuer_thread+0x570/0x570 [ 3808.328100] kthread+0x1d1/0x200 [ 3808.329213] ? set_kthread_struct+0x70/0x70 [ 3808.330573] ret_from_fork+0x1f/0x30 [ 3812.054014] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 3812.063470] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3812.097012] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3812.104617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3812.108742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3812.111593] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3812.120661] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3812.127402] RAX: 000000000000458d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3812.132300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3812.136352] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3812.140136] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3812.144161] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3812.148532] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3812.153279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3812.157001] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3812.161351] Call Trace: [ 3812.163018] [ 3812.164073] ? show_regs.cold.9+0x22/0x2f [ 3812.166488] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3812.169640] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3812.174749] ? watchdog+0x40/0x40 [ 3812.177041] ? hrtimer_interrupt+0x128/0x2c0 [ 3812.179765] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3812.182399] ? apic_timer_interrupt+0xf/0x20 [ 3812.184551] [ 3812.185797] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3812.188522] do_raw_spin_lock+0x177/0x190 [ 3812.190664] _raw_spin_lock+0x21/0x30 [ 3812.192557] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3812.195408] ? do_raw_spin_unlock+0x75/0x190 [ 3812.198037] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3812.200671] ? do_wait_intr+0xf0/0xf0 [ 3812.202551] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3812.205272] mdc_close+0x53b/0xd70 [mdc] [ 3812.207292] lmv_close+0x18d/0x490 [lmv] [ 3812.209434] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3812.215047] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3812.218414] ll_md_close+0x274/0x800 [lustre] [ 3812.220592] ll_file_release+0x128/0xc50 [lustre] [ 3812.223076] __fput+0x10d/0x440 [ 3812.224519] ____fput+0x12/0x20 [ 3812.226065] task_work_run+0xc6/0x110 [ 3812.228116] exit_to_usermode_loop+0x1dd/0x1f0 [ 3812.230891] do_syscall_64+0x3d6/0x440 [ 3812.233092] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3812.240605] RIP: 0033:0x7fdc0afbfcd8 [ 3812.244627] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3812.256542] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3812.262386] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3812.265984] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3812.270050] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3812.276155] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3812.284051] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3824.048047] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3824.051160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3824.052072] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3824.068575] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3824.072171] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3824.072246] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3824.072252] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3824.072255] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3824.072267] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3824.072272] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3824.072276] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3824.072278] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3824.072281] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3824.072283] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3824.072285] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3824.072288] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3824.072291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3824.072294] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3824.072303] Call Trace: [ 3824.072309] [ 3824.072314] ? show_regs.cold.9+0x22/0x2f [ 3824.072323] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3824.072328] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3824.072333] ? watchdog+0x40/0x40 [ 3824.072337] ? hrtimer_interrupt+0x128/0x2c0 [ 3824.072340] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3824.072347] ? apic_timer_interrupt+0xf/0x20 [ 3824.072351] [ 3824.072354] ? native_safe_halt+0x2e/0x50 [ 3824.072358] kvm_wait+0xcd/0xe0 [ 3824.072365] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3824.072371] do_raw_spin_lock+0x177/0x190 [ 3824.072375] _raw_spin_lock+0x21/0x30 [ 3824.072380] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3824.079836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3824.079846] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3824.079861] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3824.079866] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3824.079871] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3824.079873] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3824.079875] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3824.079877] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3824.079879] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3824.079882] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3824.079884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3824.079887] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3824.079906] Call Trace: [ 3824.079913] [ 3824.079918] ? show_regs.cold.9+0x22/0x2f [ 3824.079931] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3824.079939] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3824.079946] ? watchdog+0x40/0x40 [ 3824.079951] ? hrtimer_interrupt+0x128/0x2c0 [ 3824.080078] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3824.080089] ? apic_timer_interrupt+0xf/0x20 [ 3824.080096] [ 3824.080099] ? native_safe_halt+0x2e/0x50 [ 3824.080104] kvm_wait+0xcd/0xe0 [ 3824.080113] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3824.080121] ? dump_stack+0xf6/0x10e [ 3824.080128] do_raw_spin_lock+0x177/0x190 [ 3824.080133] _raw_spin_lock+0x21/0x30 [ 3824.080141] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3824.096139] ? do_raw_spin_unlock+0x75/0x190 [ 3824.099968] obd_mod_free+0x46/0x140 [ptlrpc] [ 3824.102606] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3824.104152] mdc_commit_open+0x91/0xb0 [mdc] [ 3824.112988] ? do_wait_intr+0xf0/0xf0 [ 3824.113010] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3824.117143] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3824.121928] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3824.122222] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3824.122572] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3824.122686] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3824.122726] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3824.122757] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3824.122786] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3824.122873] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3824.123109] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3824.123140] lmv_intent_open+0x338/0x1150 [lmv] [ 3824.123173] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3824.123261] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3824.123348] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3824.123372] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3824.123585] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3824.130082] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3824.130379] after_reply+0xe2c/0x1780 [ptlrpc] [ 3824.130586] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3824.130797] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3824.131023] ? do_wait_intr+0xf0/0xf0 [ 3824.131034] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3824.131243] kthread+0x1d1/0x200 [ 3824.131250] ? set_kthread_struct+0x70/0x70 [ 3824.131254] ret_from_fork+0x1f/0x30 [ 3824.365949] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3824.367681] ? do_raw_spin_unlock+0x75/0x190 [ 3824.369642] ? _raw_spin_unlock+0x12/0x30 [ 3824.371249] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3824.373459] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3824.375704] ll_atomic_open+0x36a/0xd70 [lustre] [ 3824.382368] lookup_open+0x356/0xb40 [ 3824.384346] do_last+0x313/0xfc0 [ 3824.386190] ? nd_jump_root+0xe5/0x160 [ 3824.388953] ? path_init+0x437/0x520 [ 3824.391815] path_openat+0xf7/0x500 [ 3824.393236] ? do_raw_spin_unlock+0x75/0x190 [ 3824.395193] do_filp_open+0x99/0x140 [ 3824.400460] ? getname_flags+0x6e/0x330 [ 3824.405055] ? __check_object_size+0xff/0x256 [ 3824.407371] ? do_raw_spin_unlock+0x75/0x190 [ 3824.409631] ? _raw_spin_unlock+0x12/0x30 [ 3824.411620] do_sys_openat2+0x2b4/0x410 [ 3824.413914] do_sys_open+0x73/0xa0 [ 3824.415396] __x64_sys_openat+0x24/0x30 [ 3824.417599] do_syscall_64+0xc1/0x440 [ 3824.419845] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3824.421801] RIP: 0033:0x7fdc0afbf332 [ 3824.425244] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3824.437213] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3824.442146] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3824.447964] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3824.451141] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3824.454124] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3824.456793] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3829.216228] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1804s! [ 3829.220853] Showing busy workqueues and worker pools: [ 3829.223841] workqueue events: flags=0x0 [ 3829.226763] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3829.231700] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3829.235215] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3829.238364] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3829.243787] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3829.248380] pending: kfree_rcu_monitor [ 3829.250724] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3829.255044] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 3829.261835] workqueue events_long: flags=0x0 [ 3829.264249] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3829.267608] pending: lu_objects_flush [obdclass] [ 3829.271169] workqueue events_unbound: flags=0x2 [ 3829.272906] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3829.277212] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3829.284653] workqueue events_power_efficient: flags=0x80 [ 3829.287284] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3829.290384] pending: neigh_periodic_work, check_lifetime [ 3829.292768] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3829.298150] pending: do_cache_clean, neigh_periodic_work [ 3829.308080] workqueue mm_percpu_wq: flags=0x8 [ 3829.310745] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3829.315874] pending: lru_add_drain_per_cpu [ 3829.320342] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3829.324211] pending: vmstat_update, lru_add_drain_per_cpu [ 3829.328446] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3829.338521] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3829.341917] workqueue writeback: flags=0x4a [ 3829.343979] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3829.347950] pending: wb_workfn, wb_workfn [ 3829.350095] workqueue xprtiod: flags=0xa [ 3829.352026] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3829.356035] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3829.360516] workqueue ipv6_addrconf: flags=0x40008 [ 3829.364041] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3829.367637] pending: addrconf_verify_work [ 3829.369751] workqueue knbd0-recv: flags=0x1a [ 3829.372099] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3829.375690] in-flight: 655:recv_work [ 3829.377443] workqueue ptlrpc_pinger: flags=0x2 [ 3829.379088] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3829.381211] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3829.383757] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1804s workers=4 idle: 36396 38 11188 [ 3829.386732] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3819s workers=2 idle: 658 [ 3836.056113] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3836.061688] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3836.134870] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3836.161484] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3836.172475] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3836.178444] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3836.181662] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3836.197534] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3836.203238] RAX: 0000000000002f92 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3836.209501] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3836.220278] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3836.232430] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3836.240956] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3836.247036] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3836.254231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3836.260671] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3836.267508] Call Trace: [ 3836.268350] [ 3836.271690] ? show_regs.cold.9+0x22/0x2f [ 3836.274510] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3836.279722] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3836.283492] ? watchdog+0x40/0x40 [ 3836.286058] ? hrtimer_interrupt+0x128/0x2c0 [ 3836.290326] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3836.294695] ? apic_timer_interrupt+0xf/0x20 [ 3836.298032] [ 3836.300644] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3836.302660] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3836.304673] do_raw_spin_lock+0x177/0x190 [ 3836.307561] _raw_spin_lock+0x21/0x30 [ 3836.314336] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3836.320491] process_one_work+0x2c8/0x700 [ 3836.325757] worker_thread+0x296/0x6e0 [ 3836.328638] ? rescuer_thread+0x570/0x570 [ 3836.331621] kthread+0x1d1/0x200 [ 3836.333464] ? set_kthread_struct+0x70/0x70 [ 3836.335714] ret_from_fork+0x1f/0x30 [ 3840.054101] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 3840.057430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3840.077339] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3840.082399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3840.085850] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3840.088732] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3840.097545] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3840.100711] RAX: 000000000000176c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3840.104472] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3840.107843] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3840.111108] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3840.114352] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3840.117481] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3840.121763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3840.125405] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3840.128019] Call Trace: [ 3840.130274] [ 3840.130856] ? show_regs.cold.9+0x22/0x2f [ 3840.131979] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3840.134651] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3840.136650] ? watchdog+0x40/0x40 [ 3840.138733] ? hrtimer_interrupt+0x128/0x2c0 [ 3840.140796] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3840.143046] ? apic_timer_interrupt+0xf/0x20 [ 3840.144688] [ 3840.145961] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3840.148209] do_raw_spin_lock+0x177/0x190 [ 3840.149850] _raw_spin_lock+0x21/0x30 [ 3840.151637] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3840.154574] ? do_raw_spin_unlock+0x75/0x190 [ 3840.156727] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3840.159455] ? do_wait_intr+0xf0/0xf0 [ 3840.160946] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3840.164488] mdc_close+0x53b/0xd70 [mdc] [ 3840.166276] lmv_close+0x18d/0x490 [lmv] [ 3840.168280] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3840.170933] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3840.173725] ll_md_close+0x274/0x800 [lustre] [ 3840.176262] ll_file_release+0x128/0xc50 [lustre] [ 3840.178575] __fput+0x10d/0x440 [ 3840.180105] ____fput+0x12/0x20 [ 3840.181321] task_work_run+0xc6/0x110 [ 3840.183153] exit_to_usermode_loop+0x1dd/0x1f0 [ 3840.185297] do_syscall_64+0x3d6/0x440 [ 3840.187467] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3840.189947] RIP: 0033:0x7fdc0afbfcd8 [ 3840.191677] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3840.203521] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3840.212361] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3840.217345] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3840.221169] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3840.224864] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3840.228258] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3852.052009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3852.054998] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3852.075848] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3852.080951] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3852.085185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3852.087973] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3852.096843] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3852.100643] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3852.103853] RDX: 00000000000004ca RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3852.107239] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3852.110668] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3852.113906] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3852.117328] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3852.120969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3852.123746] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3852.126833] Call Trace: [ 3852.128005] [ 3852.128931] ? show_regs.cold.9+0x22/0x2f [ 3852.130810] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3852.132866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3852.135103] ? watchdog+0x40/0x40 [ 3852.136804] ? hrtimer_interrupt+0x128/0x2c0 [ 3852.138488] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3852.140654] ? apic_timer_interrupt+0xf/0x20 [ 3852.142524] [ 3852.143648] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3852.146146] do_raw_spin_lock+0x177/0x190 [ 3852.147946] _raw_spin_lock+0x21/0x30 [ 3852.149885] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3852.152757] ? do_raw_spin_unlock+0x75/0x190 [ 3852.154809] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3852.157467] ? do_wait_intr+0xf0/0xf0 [ 3852.159178] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3852.161357] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3852.163621] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3852.166481] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3852.168197] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3852.170850] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3852.172985] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3852.175122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3852.177818] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3852.180887] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3852.183452] lmv_intent_open+0x338/0x1150 [lmv] [ 3852.185685] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3852.188394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3852.191241] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3852.193385] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3852.195853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3852.198637] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3852.200928] ? do_raw_spin_unlock+0x75/0x190 [ 3852.202974] ? _raw_spin_unlock+0x12/0x30 [ 3852.204788] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3852.207310] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3852.209774] ll_atomic_open+0x36a/0xd70 [lustre] [ 3852.212201] lookup_open+0x356/0xb40 [ 3852.213712] do_last+0x313/0xfc0 [ 3852.214751] ? nd_jump_root+0xe5/0x160 [ 3852.215894] ? path_init+0x437/0x520 [ 3852.217457] path_openat+0xf7/0x500 [ 3852.218979] ? do_raw_spin_unlock+0x75/0x190 [ 3852.221207] do_filp_open+0x99/0x140 [ 3852.222842] ? getname_flags+0x6e/0x330 [ 3852.224743] ? __check_object_size+0xff/0x256 [ 3852.226637] ? do_raw_spin_unlock+0x75/0x190 [ 3852.228450] ? _raw_spin_unlock+0x12/0x30 [ 3852.230155] do_sys_openat2+0x2b4/0x410 [ 3852.232234] do_sys_open+0x73/0xa0 [ 3852.233824] __x64_sys_openat+0x24/0x30 [ 3852.236361] do_syscall_64+0xc1/0x440 [ 3852.238106] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3852.240539] RIP: 0033:0x7fdc0afbf332 [ 3852.242317] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3852.312210] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3852.320083] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3852.330126] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3852.345211] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3852.354438] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3852.369062] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3856.048283] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 3856.057013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3856.101267] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3856.108542] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3856.116532] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3856.120207] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3856.133999] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3856.140306] RAX: 0000000000006731 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3856.142769] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3856.153529] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3856.163802] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3856.167222] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3856.175572] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3856.185617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3856.190244] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3856.198971] Call Trace: [ 3856.199687] [ 3856.200247] ? show_regs.cold.9+0x22/0x2f [ 3856.202078] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3856.204571] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3856.209108] ? watchdog+0x40/0x40 [ 3856.210984] ? hrtimer_interrupt+0x128/0x2c0 [ 3856.212754] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3856.214246] ? apic_timer_interrupt+0xf/0x20 [ 3856.217197] [ 3856.220207] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3856.228082] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3856.230554] ? dump_stack+0xf6/0x10e [ 3856.232257] do_raw_spin_lock+0x177/0x190 [ 3856.233977] _raw_spin_lock+0x21/0x30 [ 3856.235851] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3856.243746] obd_mod_free+0x46/0x140 [ptlrpc] [ 3856.248809] mdc_commit_open+0x91/0xb0 [mdc] [ 3856.250774] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3856.255181] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3856.259050] after_reply+0xe2c/0x1780 [ptlrpc] [ 3856.267119] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3856.273464] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3856.282045] ? do_wait_intr+0xf0/0xf0 [ 3856.288420] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3856.294764] kthread+0x1d1/0x200 [ 3856.295931] ? set_kthread_struct+0x70/0x70 [ 3856.297128] ret_from_fork+0x1f/0x30 [ 3859.423047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1834s! [ 3859.425622] Showing busy workqueues and worker pools: [ 3859.427838] workqueue events: flags=0x0 [ 3859.433554] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3859.447164] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3859.454181] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3859.460828] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3859.468149] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3859.473156] pending: kfree_rcu_monitor [ 3859.475537] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3859.480294] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 3859.489130] workqueue events_long: flags=0x0 [ 3859.493793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3859.501503] pending: lu_objects_flush [obdclass] [ 3859.506262] workqueue events_unbound: flags=0x2 [ 3859.508823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3859.519729] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3859.531644] workqueue events_power_efficient: flags=0x80 [ 3859.536196] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3859.542164] pending: neigh_periodic_work, check_lifetime [ 3859.544602] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3859.548650] pending: do_cache_clean, neigh_periodic_work [ 3859.552101] workqueue mm_percpu_wq: flags=0x8 [ 3859.554387] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3859.557673] pending: lru_add_drain_per_cpu [ 3859.560564] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3859.571376] pending: vmstat_update, lru_add_drain_per_cpu [ 3859.576450] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3859.582228] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3859.588673] workqueue writeback: flags=0x4a [ 3859.592185] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3859.595082] pending: wb_workfn, wb_workfn [ 3859.598457] workqueue xprtiod: flags=0xa [ 3859.604339] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3859.610849] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3859.615419] workqueue ipv6_addrconf: flags=0x40008 [ 3859.621230] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3859.626512] pending: addrconf_verify_work [ 3859.627961] workqueue knbd0-recv: flags=0x1a [ 3859.631153] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3859.634617] in-flight: 655:recv_work [ 3859.637387] workqueue ptlrpc_pinger: flags=0x2 [ 3859.639201] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3859.641632] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3859.645132] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1834s workers=4 idle: 36396 38 11188 [ 3859.648163] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3850s workers=2 idle: 658 [ 3864.055093] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3864.063542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3864.091614] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3864.106112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3864.110264] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3864.115813] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3864.117800] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3864.131322] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3864.138666] RAX: 0000000000003d90 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3864.141278] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3864.143750] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3864.149633] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3864.155915] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3864.162663] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3864.167472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3864.173262] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3864.177952] Call Trace: [ 3864.179766] [ 3864.181673] ? show_regs.cold.9+0x22/0x2f [ 3864.184589] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3864.189296] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3864.191172] ? watchdog+0x40/0x40 [ 3864.192447] ? hrtimer_interrupt+0x128/0x2c0 [ 3864.195245] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3864.196956] ? apic_timer_interrupt+0xf/0x20 [ 3864.200520] [ 3864.202573] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3864.206714] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3864.211316] do_raw_spin_lock+0x177/0x190 [ 3864.212739] _raw_spin_lock+0x21/0x30 [ 3864.214440] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3864.221751] process_one_work+0x2c8/0x700 [ 3864.224097] worker_thread+0x296/0x6e0 [ 3864.225331] ? rescuer_thread+0x570/0x570 [ 3864.229280] kthread+0x1d1/0x200 [ 3864.230590] ? set_kthread_struct+0x70/0x70 [ 3864.237160] ret_from_fork+0x1f/0x30 [ 3868.054033] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3868.057411] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3868.078921] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3868.083679] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3868.088237] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3868.091435] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3868.101496] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3868.105028] RAX: 0000000000005962 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3868.108436] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3868.111746] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3868.115516] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3868.119776] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3868.123202] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3868.126974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3868.130176] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3868.133684] Call Trace: [ 3868.134706] [ 3868.135746] ? show_regs.cold.9+0x22/0x2f [ 3868.140513] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3868.143095] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3868.145150] ? watchdog+0x40/0x40 [ 3868.146625] ? hrtimer_interrupt+0x128/0x2c0 [ 3868.148583] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3868.150804] ? apic_timer_interrupt+0xf/0x20 [ 3868.152587] [ 3868.153430] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3868.156062] do_raw_spin_lock+0x177/0x190 [ 3868.157575] _raw_spin_lock+0x21/0x30 [ 3868.159237] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3868.161543] ? do_raw_spin_unlock+0x75/0x190 [ 3868.163305] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3868.165495] ? do_wait_intr+0xf0/0xf0 [ 3868.167090] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3868.169261] mdc_close+0x53b/0xd70 [mdc] [ 3868.171075] lmv_close+0x18d/0x490 [lmv] [ 3868.172799] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3868.178288] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3868.181370] ll_md_close+0x274/0x800 [lustre] [ 3868.184412] ll_file_release+0x128/0xc50 [lustre] [ 3868.186244] __fput+0x10d/0x440 [ 3868.187525] ____fput+0x12/0x20 [ 3868.188905] task_work_run+0xc6/0x110 [ 3868.190292] exit_to_usermode_loop+0x1dd/0x1f0 [ 3868.192197] do_syscall_64+0x3d6/0x440 [ 3868.193492] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3868.195584] RIP: 0033:0x7fdc0afbfcd8 [ 3868.197195] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3868.212042] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3868.215131] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3868.218185] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3868.221273] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3868.224556] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3868.226803] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3879.385082] rcu: INFO: rcu_sched self-detected stall on CPU [ 3879.388559] rcu: 0-...!: (1073827 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 3879.394688] (t=1860042 jiffies g=270761 q=3169) [ 3879.398644] rcu: rcu_sched kthread starved for 1854974 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 3879.412403] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3879.419428] rcu: RCU grace-period kthread stack dump: [ 3879.422586] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3879.429504] Call Trace: [ 3879.431366] __schedule+0x351/0xcb0 [ 3879.432263] schedule+0xc0/0x180 [ 3879.434143] schedule_timeout+0xb4/0x190 [ 3879.435869] ? do_raw_spin_unlock+0x75/0x190 [ 3879.438929] ? __next_timer_interrupt+0x160/0x160 [ 3879.441395] ? prepare_to_swait_event+0xf3/0x140 [ 3879.445503] rcu_gp_kthread+0x8dd/0x15b0 [ 3879.450278] ? force_qs_rnp+0x2c0/0x2c0 [ 3879.452568] kthread+0x1d1/0x200 [ 3879.456626] ? set_kthread_struct+0x70/0x70 [ 3879.460148] ret_from_fork+0x1f/0x30 [ 3879.463627] rcu: Stack dump where RCU GP kthread last ran: [ 3879.467280] Sending NMI from CPU 0 to CPUs 3: [ 3879.475420] NMI backtrace for cpu 3 [ 3879.475433] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3879.475436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3879.475437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3879.475439] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3879.475442] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000216 [ 3879.475444] RAX: 0000000000004c0a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3879.475446] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3879.475447] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3879.475449] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3879.475450] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3879.475452] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3879.475453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3879.475454] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3879.475456] Call Trace: [ 3879.475456] [ 3879.475457] ? show_regs.cold.9+0x22/0x2f [ 3879.475458] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3879.475459] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3879.475460] ? nmi_handle+0xbd/0x210 [ 3879.475461] ? default_do_nmi+0x4b/0x1b0 [ 3879.475463] ? do_nmi+0x280/0x380 [ 3879.475464] ? end_repeat_nmi+0x16/0x1f [ 3879.475465] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3879.475466] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3879.475468] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3879.475469] [ 3879.475470] do_raw_spin_lock+0x177/0x190 [ 3879.475471] _raw_spin_lock+0x21/0x30 [ 3879.475472] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3879.475473] ? do_raw_spin_unlock+0x75/0x190 [ 3879.475474] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3879.475475] ? do_wait_intr+0xf0/0xf0 [ 3879.475476] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3879.475477] mdc_close+0x53b/0xd70 [mdc] [ 3879.475478] lmv_close+0x18d/0x490 [lmv] [ 3879.475479] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3879.475480] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3879.475481] ll_md_close+0x274/0x800 [lustre] [ 3879.475482] ll_file_release+0x128/0xc50 [lustre] [ 3879.475483] __fput+0x10d/0x440 [ 3879.475484] ____fput+0x12/0x20 [ 3879.475484] task_work_run+0xc6/0x110 [ 3879.475485] exit_to_usermode_loop+0x1dd/0x1f0 [ 3879.475486] do_syscall_64+0x3d6/0x440 [ 3879.475487] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3879.475488] RIP: 0033:0x7fdc0afbfcd8 [ 3879.475489] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3879.475491] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3879.475494] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3879.475495] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3879.475496] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3879.475497] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3879.475498] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3879.475654] NMI backtrace for cpu 0 [ 3879.737913] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3879.745985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3879.751458] Call Trace: [ 3879.752648] [ 3879.758406] ? dump_stack+0xbb/0x10e [ 3879.760198] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3879.764391] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3879.767492] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3879.773434] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3879.779309] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3879.784130] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3879.787592] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3879.794793] ? update_process_times+0x6d/0xc0 [ 3879.795824] ? tick_sched_handle.isra.18+0x27/0x90 [ 3879.801820] ? tick_sched_timer+0x53/0xb0 [ 3879.806130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3879.810907] ? tick_sched_handle.isra.18+0x90/0x90 [ 3879.817638] ? hrtimer_interrupt+0x128/0x2c0 [ 3879.823697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3879.825070] ? apic_timer_interrupt+0xf/0x20 [ 3879.830290] [ 3879.831326] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3879.841480] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3879.843590] ? dump_stack+0xf6/0x10e [ 3879.844739] ? do_raw_spin_lock+0x177/0x190 [ 3879.854387] ? _raw_spin_lock+0x21/0x30 [ 3879.860898] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3879.870987] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3879.874475] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3879.876967] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3879.887112] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3879.895449] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3879.900165] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3879.909067] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3879.911943] ? do_wait_intr+0xf0/0xf0 [ 3879.913393] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3879.917142] ? kthread+0x1d1/0x200 [ 3879.918502] ? set_kthread_struct+0x70/0x70 [ 3879.920823] ? ret_from_fork+0x1f/0x30 [ 3880.052007] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3880.058417] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3880.084871] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3880.091372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3880.096019] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3880.099238] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3880.109898] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3880.114465] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3880.119354] RDX: 00000000000054cd RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3880.123468] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3880.127137] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3880.131322] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3880.135801] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3880.140793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3880.144303] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3880.148112] Call Trace: [ 3880.149398] [ 3880.150894] ? show_regs.cold.9+0x22/0x2f [ 3880.153084] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3880.159233] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3880.164202] ? watchdog+0x40/0x40 [ 3880.165452] ? hrtimer_interrupt+0x128/0x2c0 [ 3880.170644] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3880.174134] ? apic_timer_interrupt+0xf/0x20 [ 3880.177519] [ 3880.178296] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3880.183465] do_raw_spin_lock+0x177/0x190 [ 3880.188339] _raw_spin_lock+0x21/0x30 [ 3880.191093] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3880.197894] ? do_raw_spin_unlock+0x75/0x190 [ 3880.202914] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3880.207871] ? do_wait_intr+0xf0/0xf0 [ 3880.211040] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3880.215817] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3880.219695] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3880.223675] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3880.226737] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3880.229983] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3880.234014] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3880.237859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3880.241738] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3880.245521] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3880.253288] lmv_intent_open+0x338/0x1150 [lmv] [ 3880.256284] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3880.261969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3880.263953] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3880.268244] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3880.275287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3880.277409] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3880.278740] ? do_raw_spin_unlock+0x75/0x190 [ 3880.283604] ? _raw_spin_unlock+0x12/0x30 [ 3880.285965] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3880.291872] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3880.297494] ll_atomic_open+0x36a/0xd70 [lustre] [ 3880.299361] lookup_open+0x356/0xb40 [ 3880.301492] do_last+0x313/0xfc0 [ 3880.303584] ? nd_jump_root+0xe5/0x160 [ 3880.310822] ? path_init+0x437/0x520 [ 3880.312738] path_openat+0xf7/0x500 [ 3880.316823] ? do_raw_spin_unlock+0x75/0x190 [ 3880.318397] do_filp_open+0x99/0x140 [ 3880.321287] ? getname_flags+0x6e/0x330 [ 3880.324353] ? __check_object_size+0xff/0x256 [ 3880.327286] ? do_raw_spin_unlock+0x75/0x190 [ 3880.331639] ? _raw_spin_unlock+0x12/0x30 [ 3880.334659] do_sys_openat2+0x2b4/0x410 [ 3880.339606] do_sys_open+0x73/0xa0 [ 3880.342006] __x64_sys_openat+0x24/0x30 [ 3880.343833] do_syscall_64+0xc1/0x440 [ 3880.345688] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3880.348734] RIP: 0033:0x7fdc0afbf332 [ 3880.351479] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3880.371243] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3880.384776] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3880.392008] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3880.401311] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3880.410687] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3880.419597] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3890.143040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1865s! [ 3890.145630] Showing busy workqueues and worker pools: [ 3890.147502] workqueue events: flags=0x0 [ 3890.148761] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3890.151023] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3890.153171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3890.155915] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3890.161013] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3890.164684] pending: kfree_rcu_monitor [ 3890.166854] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3890.170784] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 3890.176898] workqueue events_long: flags=0x0 [ 3890.178750] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3890.181050] pending: lu_objects_flush [obdclass] [ 3890.182666] workqueue events_unbound: flags=0x2 [ 3890.184557] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3890.187630] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3890.194810] workqueue events_power_efficient: flags=0x80 [ 3890.198209] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3890.202131] pending: neigh_periodic_work, check_lifetime [ 3890.205831] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3890.209340] pending: do_cache_clean, neigh_periodic_work [ 3890.211627] workqueue mm_percpu_wq: flags=0x8 [ 3890.213235] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3890.216123] pending: lru_add_drain_per_cpu [ 3890.218413] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3890.221902] pending: vmstat_update, lru_add_drain_per_cpu [ 3890.229580] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3890.237236] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3890.242481] workqueue writeback: flags=0x4a [ 3890.247144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3890.254186] pending: wb_workfn, wb_workfn [ 3890.258966] workqueue xprtiod: flags=0xa [ 3890.261237] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3890.263668] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3890.269804] workqueue ipv6_addrconf: flags=0x40008 [ 3890.271418] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3890.277288] pending: addrconf_verify_work [ 3890.281363] workqueue knbd0-recv: flags=0x1a [ 3890.284827] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3890.289183] in-flight: 655:recv_work [ 3890.291574] workqueue ptlrpc_pinger: flags=0x2 [ 3890.293307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3890.297304] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3890.299698] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1865s workers=4 idle: 36396 38 11188 [ 3890.305885] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3880s workers=2 idle: 658 [ 3892.051045] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3892.053593] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3892.086763] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3892.090925] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3892.096105] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3892.099212] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3892.100927] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3892.109791] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3892.113398] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3892.116570] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3892.120545] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3892.124508] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3892.128250] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3892.132432] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3892.135586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3892.137508] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3892.140833] Call Trace: [ 3892.143053] [ 3892.144011] ? show_regs.cold.9+0x22/0x2f [ 3892.145532] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3892.147324] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3892.149011] ? watchdog+0x40/0x40 [ 3892.150350] ? hrtimer_interrupt+0x128/0x2c0 [ 3892.151688] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3892.153563] ? apic_timer_interrupt+0xf/0x20 [ 3892.155428] [ 3892.156159] ? native_safe_halt+0x2e/0x50 [ 3892.157728] kvm_wait+0xcd/0xe0 [ 3892.159014] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3892.161487] do_raw_spin_lock+0x177/0x190 [ 3892.162598] _raw_spin_lock+0x21/0x30 [ 3892.163974] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3892.166641] process_one_work+0x2c8/0x700 [ 3892.169461] worker_thread+0x296/0x6e0 [ 3892.170892] ? rescuer_thread+0x570/0x570 [ 3892.172394] kthread+0x1d1/0x200 [ 3892.173430] ? set_kthread_struct+0x70/0x70 [ 3892.175384] ret_from_fork+0x1f/0x30 [ 3904.054054] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 3904.059400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3904.098362] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3904.107049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3904.115135] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3904.116870] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3904.123815] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3904.126321] RAX: 000000000000656f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3904.128204] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3904.130738] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3904.144965] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3904.148656] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3904.152133] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3904.156005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3904.158498] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3904.161674] Call Trace: [ 3904.165285] [ 3904.166523] ? show_regs.cold.9+0x22/0x2f [ 3904.168585] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3904.171721] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3904.173963] ? watchdog+0x40/0x40 [ 3904.175640] ? hrtimer_interrupt+0x128/0x2c0 [ 3904.177539] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3904.179062] ? apic_timer_interrupt+0xf/0x20 [ 3904.181584] [ 3904.182267] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3904.185044] do_raw_spin_lock+0x177/0x190 [ 3904.187290] _raw_spin_lock+0x21/0x30 [ 3904.188968] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3904.192537] ? do_raw_spin_unlock+0x75/0x190 [ 3904.194885] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3904.197952] ? do_wait_intr+0xf0/0xf0 [ 3904.199661] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3904.202769] mdc_close+0x53b/0xd70 [mdc] [ 3904.204666] lmv_close+0x18d/0x490 [lmv] [ 3904.206552] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3904.209388] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3904.211692] ll_md_close+0x274/0x800 [lustre] [ 3904.213959] ll_file_release+0x128/0xc50 [lustre] [ 3904.216936] __fput+0x10d/0x440 [ 3904.218463] ____fput+0x12/0x20 [ 3904.219982] task_work_run+0xc6/0x110 [ 3904.222469] exit_to_usermode_loop+0x1dd/0x1f0 [ 3904.225524] do_syscall_64+0x3d6/0x440 [ 3904.229852] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3904.238970] RIP: 0033:0x7fdc0afbfcd8 [ 3904.242283] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3904.263670] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3904.269692] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3904.276574] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3904.280967] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3904.285008] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3904.288837] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3908.053178] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3908.060049] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3908.092535] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3908.096394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3908.101277] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3908.107329] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3908.122746] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3908.125881] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3908.130761] RDX: 0000000000003388 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3908.134225] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3908.138625] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3908.142665] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3908.144551] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3908.151005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3908.153411] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3908.155597] Call Trace: [ 3908.156544] [ 3908.159453] ? show_regs.cold.9+0x22/0x2f [ 3908.161454] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3908.168544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3908.170694] ? watchdog+0x40/0x40 [ 3908.172274] ? hrtimer_interrupt+0x128/0x2c0 [ 3908.174498] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3908.176768] ? apic_timer_interrupt+0xf/0x20 [ 3908.179591] [ 3908.180184] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3908.182033] do_raw_spin_lock+0x177/0x190 [ 3908.188302] _raw_spin_lock+0x21/0x30 [ 3908.191739] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3908.194613] ? do_raw_spin_unlock+0x75/0x190 [ 3908.195876] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3908.199137] ? do_wait_intr+0xf0/0xf0 [ 3908.202931] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3908.205703] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3908.207349] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3908.212883] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3908.214388] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3908.219583] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3908.222174] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3908.224596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3908.227875] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3908.236438] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3908.238993] lmv_intent_open+0x338/0x1150 [lmv] [ 3908.241276] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3908.246370] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3908.247804] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3908.249109] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3908.251348] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3908.252750] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3908.255634] ? do_raw_spin_unlock+0x75/0x190 [ 3908.256760] ? _raw_spin_unlock+0x12/0x30 [ 3908.258778] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3908.261805] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3908.265961] ll_atomic_open+0x36a/0xd70 [lustre] [ 3908.269865] lookup_open+0x356/0xb40 [ 3908.273734] do_last+0x313/0xfc0 [ 3908.274917] ? nd_jump_root+0xe5/0x160 [ 3908.276033] ? path_init+0x437/0x520 [ 3908.278578] path_openat+0xf7/0x500 [ 3908.280412] ? do_raw_spin_unlock+0x75/0x190 [ 3908.283703] do_filp_open+0x99/0x140 [ 3908.285786] ? getname_flags+0x6e/0x330 [ 3908.288506] ? __check_object_size+0xff/0x256 [ 3908.289987] ? do_raw_spin_unlock+0x75/0x190 [ 3908.292758] ? _raw_spin_unlock+0x12/0x30 [ 3908.294881] do_sys_openat2+0x2b4/0x410 [ 3908.296691] do_sys_open+0x73/0xa0 [ 3908.297923] __x64_sys_openat+0x24/0x30 [ 3908.299353] do_syscall_64+0xc1/0x440 [ 3908.300471] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3908.303309] RIP: 0033:0x7fdc0afbf332 [ 3908.304882] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3908.312663] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3908.316624] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3908.320024] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3908.323897] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3908.326788] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3908.330296] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3916.052510] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 3916.058797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3916.115167] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3916.128141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3916.136262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3916.138751] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3916.160369] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3916.174065] RAX: 0000000000007b86 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3916.177295] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3916.182665] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3916.189467] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3916.191990] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3916.194597] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3916.203482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3916.206566] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3916.212148] Call Trace: [ 3916.213636] [ 3916.214264] ? show_regs.cold.9+0x22/0x2f [ 3916.215674] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3916.219065] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3916.221672] ? watchdog+0x40/0x40 [ 3916.222552] ? hrtimer_interrupt+0x128/0x2c0 [ 3916.223526] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3916.229572] ? apic_timer_interrupt+0xf/0x20 [ 3916.231707] [ 3916.232489] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3916.241662] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3916.247516] ? dump_stack+0xf6/0x10e [ 3916.253378] do_raw_spin_lock+0x177/0x190 [ 3916.259614] _raw_spin_lock+0x21/0x30 [ 3916.261695] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3916.266667] obd_mod_free+0x46/0x140 [ptlrpc] [ 3916.273459] mdc_commit_open+0x91/0xb0 [mdc] [ 3916.276795] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3916.281356] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3916.290108] after_reply+0xe2c/0x1780 [ptlrpc] [ 3916.293037] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3916.296683] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3916.301688] ? do_wait_intr+0xf0/0xf0 [ 3916.303051] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3916.307832] kthread+0x1d1/0x200 [ 3916.310362] ? set_kthread_struct+0x70/0x70 [ 3916.312655] ret_from_fork+0x1f/0x30 [ 3920.051058] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3920.055619] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3920.083382] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3920.089276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3920.093457] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3920.097127] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3920.100731] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3920.110201] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3920.113995] RAX: 0000000000004c81 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3920.118375] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3920.122148] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3920.125357] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3920.129219] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3920.132863] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3920.137921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3920.140074] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3920.143175] Call Trace: [ 3920.144422] [ 3920.146532] ? show_regs.cold.9+0x22/0x2f [ 3920.148420] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3920.154476] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3920.158242] ? watchdog+0x40/0x40 [ 3920.159700] ? hrtimer_interrupt+0x128/0x2c0 [ 3920.162074] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3920.164747] ? apic_timer_interrupt+0xf/0x20 [ 3920.166868] [ 3920.168034] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3920.171227] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3920.173973] do_raw_spin_lock+0x177/0x190 [ 3920.176158] _raw_spin_lock+0x21/0x30 [ 3920.178464] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3920.182420] process_one_work+0x2c8/0x700 [ 3920.184944] worker_thread+0x296/0x6e0 [ 3920.187273] ? rescuer_thread+0x570/0x570 [ 3920.189159] kthread+0x1d1/0x200 [ 3920.190789] ? set_kthread_struct+0x70/0x70 [ 3920.192976] ret_from_fork+0x1f/0x30 [ 3920.351074] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1895s! [ 3920.360531] Showing busy workqueues and worker pools: [ 3920.366646] workqueue events: flags=0x0 [ 3920.368431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3920.371671] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3920.374769] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3920.384008] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3920.391142] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3920.402952] pending: kfree_rcu_monitor [ 3920.410473] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3920.425408] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 3920.435107] workqueue events_long: flags=0x0 [ 3920.441465] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3920.448615] pending: lu_objects_flush [obdclass] [ 3920.451397] workqueue events_unbound: flags=0x2 [ 3920.453239] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3920.456229] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3920.462570] workqueue events_power_efficient: flags=0x80 [ 3920.468446] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3920.469987] pending: neigh_periodic_work, check_lifetime [ 3920.476516] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3920.478831] pending: do_cache_clean, neigh_periodic_work [ 3920.487232] workqueue mm_percpu_wq: flags=0x8 [ 3920.493514] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3920.507692] pending: lru_add_drain_per_cpu [ 3920.512251] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3920.525371] pending: vmstat_update, lru_add_drain_per_cpu [ 3920.532504] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3920.535513] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3920.541623] workqueue writeback: flags=0x4a [ 3920.547150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3920.554399] pending: wb_workfn, wb_workfn [ 3920.556016] workqueue xprtiod: flags=0xa [ 3920.568017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3920.570405] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3920.574417] workqueue ipv6_addrconf: flags=0x40008 [ 3920.581663] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3920.588378] pending: addrconf_verify_work [ 3920.590699] workqueue knbd0-recv: flags=0x1a [ 3920.592499] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3920.601557] in-flight: 655:recv_work [ 3920.607167] workqueue ptlrpc_pinger: flags=0x2 [ 3920.609671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3920.615359] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3920.619625] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1895s workers=4 idle: 36396 38 11188 [ 3920.627329] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3911s workers=2 idle: 658 [ 3932.055071] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 3932.063165] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3932.104720] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3932.109684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3932.112859] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3932.115303] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3932.123227] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3932.126255] RAX: 00000000000048ab RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3932.128916] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3932.131688] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3932.134394] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3932.137545] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3932.141507] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3932.144795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3932.146529] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3932.149408] Call Trace: [ 3932.150439] [ 3932.151337] ? show_regs.cold.9+0x22/0x2f [ 3932.157259] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3932.159521] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3932.166408] ? watchdog+0x40/0x40 [ 3932.169505] ? hrtimer_interrupt+0x128/0x2c0 [ 3932.172408] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3932.178316] ? apic_timer_interrupt+0xf/0x20 [ 3932.184314] [ 3932.187081] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3932.191586] do_raw_spin_lock+0x177/0x190 [ 3932.193143] _raw_spin_lock+0x21/0x30 [ 3932.198649] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3932.200592] ? do_raw_spin_unlock+0x75/0x190 [ 3932.201910] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3932.206352] ? do_wait_intr+0xf0/0xf0 [ 3932.210868] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3932.212809] mdc_close+0x53b/0xd70 [mdc] [ 3932.214431] lmv_close+0x18d/0x490 [lmv] [ 3932.216507] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3932.220044] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3932.221662] ll_md_close+0x274/0x800 [lustre] [ 3932.224613] ll_file_release+0x128/0xc50 [lustre] [ 3932.225882] __fput+0x10d/0x440 [ 3932.226936] ____fput+0x12/0x20 [ 3932.230011] task_work_run+0xc6/0x110 [ 3932.232095] exit_to_usermode_loop+0x1dd/0x1f0 [ 3932.235549] do_syscall_64+0x3d6/0x440 [ 3932.240033] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3932.242280] RIP: 0033:0x7fdc0afbfcd8 [ 3932.244418] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3932.255562] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3932.261436] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3932.264473] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3932.270077] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3932.282547] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3932.286601] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3936.052028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3936.056782] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3936.083842] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3936.089983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3936.095488] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3936.098473] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3936.108826] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3936.113312] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3936.116804] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3936.119452] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3936.121872] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3936.124778] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3936.127635] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3936.131279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3936.133383] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3936.136214] Call Trace: [ 3936.137164] [ 3936.137856] ? show_regs.cold.9+0x22/0x2f [ 3936.139291] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3936.140911] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3936.142772] ? watchdog+0x40/0x40 [ 3936.144165] ? hrtimer_interrupt+0x128/0x2c0 [ 3936.145849] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3936.147363] ? apic_timer_interrupt+0xf/0x20 [ 3936.148716] [ 3936.149477] ? native_safe_halt+0x2e/0x50 [ 3936.151178] kvm_wait+0xcd/0xe0 [ 3936.152345] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3936.154274] do_raw_spin_lock+0x177/0x190 [ 3936.155870] _raw_spin_lock+0x21/0x30 [ 3936.157181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3936.158904] ? do_raw_spin_unlock+0x75/0x190 [ 3936.160481] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3936.162774] ? do_wait_intr+0xf0/0xf0 [ 3936.164149] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3936.166211] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3936.168888] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3936.171636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3936.174404] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3936.176950] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3936.179158] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3936.181346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3936.183712] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3936.186587] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3936.188308] lmv_intent_open+0x338/0x1150 [lmv] [ 3936.190649] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3936.193154] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3936.195209] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3936.196731] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3936.198064] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3936.200144] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3936.202011] ? do_raw_spin_unlock+0x75/0x190 [ 3936.203496] ? _raw_spin_unlock+0x12/0x30 [ 3936.205042] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3936.207039] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3936.209079] ll_atomic_open+0x36a/0xd70 [lustre] [ 3936.210887] lookup_open+0x356/0xb40 [ 3936.212111] do_last+0x313/0xfc0 [ 3936.213342] ? nd_jump_root+0xe5/0x160 [ 3936.214805] ? path_init+0x437/0x520 [ 3936.216148] path_openat+0xf7/0x500 [ 3936.217736] ? do_raw_spin_unlock+0x75/0x190 [ 3936.219438] do_filp_open+0x99/0x140 [ 3936.221098] ? getname_flags+0x6e/0x330 [ 3936.222749] ? __check_object_size+0xff/0x256 [ 3936.224603] ? do_raw_spin_unlock+0x75/0x190 [ 3936.226407] ? _raw_spin_unlock+0x12/0x30 [ 3936.228147] do_sys_openat2+0x2b4/0x410 [ 3936.230873] do_sys_open+0x73/0xa0 [ 3936.232469] __x64_sys_openat+0x24/0x30 [ 3936.234733] do_syscall_64+0xc1/0x440 [ 3936.236450] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3936.238986] RIP: 0033:0x7fdc0afbf332 [ 3936.240584] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3936.247650] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3936.251193] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3936.255112] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3936.258414] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3936.260487] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3936.262748] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3948.057153] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3948.060416] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3948.061760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3948.061846] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3948.061852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3948.061854] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3948.061868] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3948.061872] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3948.061876] RAX: 0000000000002d01 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3948.061878] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3948.061881] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3948.061883] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3948.061885] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3948.061889] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3948.061891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3948.061894] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3948.061902] Call Trace: [ 3948.061907] [ 3948.061912] ? show_regs.cold.9+0x22/0x2f [ 3948.061921] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3948.062086] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3948.062090] ? watchdog+0x40/0x40 [ 3948.062094] ? hrtimer_interrupt+0x128/0x2c0 [ 3948.062098] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3948.062105] ? apic_timer_interrupt+0xf/0x20 [ 3948.062109] [ 3948.062113] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3948.062117] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3948.062121] ? dump_stack+0xf6/0x10e [ 3948.062127] do_raw_spin_lock+0x177/0x190 [ 3948.062132] _raw_spin_lock+0x21/0x30 [ 3948.062138] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3948.062479] obd_mod_free+0x46/0x140 [ptlrpc] [ 3948.062741] mdc_commit_open+0x91/0xb0 [mdc] [ 3948.062773] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3948.064160] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3948.065768] after_reply+0xe2c/0x1780 [ptlrpc] [ 3948.066120] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3948.066406] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3948.075336] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3948.075419] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3948.075427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3948.075432] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3948.075747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3948.075760] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3948.075763] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3948.075767] RAX: 0000000000003ba2 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3948.075769] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3948.075771] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3948.075773] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3948.075775] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3948.075778] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3948.075780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3948.075783] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3948.075791] Call Trace: [ 3948.075797] [ 3948.075802] ? show_regs.cold.9+0x22/0x2f [ 3948.075811] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3948.075817] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3948.075821] ? watchdog+0x40/0x40 [ 3948.075825] ? hrtimer_interrupt+0x128/0x2c0 [ 3948.075829] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3948.075850] ? apic_timer_interrupt+0xf/0x20 [ 3948.075855] [ 3948.075858] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3948.075862] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3948.075867] do_raw_spin_lock+0x177/0x190 [ 3948.075872] _raw_spin_lock+0x21/0x30 [ 3948.075876] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3948.076441] process_one_work+0x2c8/0x700 [ 3948.076448] worker_thread+0x296/0x6e0 [ 3948.076453] ? rescuer_thread+0x570/0x570 [ 3948.076456] kthread+0x1d1/0x200 [ 3948.076461] ? set_kthread_struct+0x70/0x70 [ 3948.076465] ret_from_fork+0x1f/0x30 [ 3948.356302] ? do_wait_intr+0xf0/0xf0 [ 3948.357639] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3948.360549] kthread+0x1d1/0x200 [ 3948.362313] ? set_kthread_struct+0x70/0x70 [ 3948.364908] ret_from_fork+0x1f/0x30 [ 3951.072446] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1926s! [ 3951.081206] Showing busy workqueues and worker pools: [ 3951.093463] workqueue events: flags=0x0 [ 3951.101236] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3951.111213] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3951.119729] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3951.135240] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3951.166364] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3951.183484] pending: kfree_rcu_monitor [ 3951.194648] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3951.211840] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 3951.234264] workqueue events_long: flags=0x0 [ 3951.237826] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3951.253220] pending: lu_objects_flush [obdclass] [ 3951.261677] workqueue events_unbound: flags=0x2 [ 3951.270211] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3951.282351] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3951.290840] workqueue events_power_efficient: flags=0x80 [ 3951.295173] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3951.299518] pending: neigh_periodic_work, check_lifetime [ 3951.302984] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3951.306762] pending: do_cache_clean, neigh_periodic_work [ 3951.309782] workqueue mm_percpu_wq: flags=0x8 [ 3951.313418] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3951.317375] pending: lru_add_drain_per_cpu [ 3951.319872] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3951.324152] pending: vmstat_update, lru_add_drain_per_cpu [ 3951.327876] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3951.332290] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3951.336891] workqueue writeback: flags=0x4a [ 3951.339506] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3951.343998] pending: wb_workfn, wb_workfn [ 3951.347140] workqueue xprtiod: flags=0xa [ 3951.349287] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3951.353692] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3951.359174] workqueue ipv6_addrconf: flags=0x40008 [ 3951.362590] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3951.366477] pending: addrconf_verify_work [ 3951.369633] workqueue knbd0-recv: flags=0x1a [ 3951.371260] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3951.374173] in-flight: 655:recv_work [ 3951.376238] workqueue ptlrpc_pinger: flags=0x2 [ 3951.378933] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3951.382741] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3951.386119] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1926s workers=4 idle: 36396 38 11188 [ 3951.390450] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3941s workers=2 idle: 658 [ 3960.054029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 3960.057886] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3960.083703] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3960.095698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3960.103465] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3960.106739] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3960.118863] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3960.128811] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3960.135495] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3960.139751] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3960.143537] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3960.154051] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3960.161625] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3960.171157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3960.181854] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3960.189798] Call Trace: [ 3960.192883] [ 3960.199538] ? show_regs.cold.9+0x22/0x2f [ 3960.202426] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3960.205345] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3960.207158] ? watchdog+0x40/0x40 [ 3960.212355] ? hrtimer_interrupt+0x128/0x2c0 [ 3960.216663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3960.218830] ? apic_timer_interrupt+0xf/0x20 [ 3960.220060] [ 3960.220782] ? native_safe_halt+0x2e/0x50 [ 3960.221933] kvm_wait+0xcd/0xe0 [ 3960.223207] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3960.225502] do_raw_spin_lock+0x177/0x190 [ 3960.227414] _raw_spin_lock+0x21/0x30 [ 3960.228959] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3960.230950] ? do_raw_spin_unlock+0x75/0x190 [ 3960.232720] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3960.235132] ? do_wait_intr+0xf0/0xf0 [ 3960.236701] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3960.239063] mdc_close+0x53b/0xd70 [mdc] [ 3960.240782] lmv_close+0x18d/0x490 [lmv] [ 3960.242521] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3960.244691] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3960.246814] ll_md_close+0x274/0x800 [lustre] [ 3960.248473] ll_file_release+0x128/0xc50 [lustre] [ 3960.251443] __fput+0x10d/0x440 [ 3960.252526] ____fput+0x12/0x20 [ 3960.254680] task_work_run+0xc6/0x110 [ 3960.256284] exit_to_usermode_loop+0x1dd/0x1f0 [ 3960.259466] do_syscall_64+0x3d6/0x440 [ 3960.261350] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3960.266372] RIP: 0033:0x7fdc0afbfcd8 [ 3960.268488] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3960.276969] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3960.281081] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3960.285333] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3960.291100] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3960.295381] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3960.299796] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3964.053800] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3964.064265] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3964.104615] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3964.115431] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3964.122699] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3964.124642] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3964.140324] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3964.147695] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3964.152860] RDX: 0000000000002dd8 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3964.160998] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3964.163284] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3964.167326] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3964.178577] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3964.184898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3964.189984] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3964.192239] Call Trace: [ 3964.204604] [ 3964.205532] ? show_regs.cold.9+0x22/0x2f [ 3964.212202] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3964.216732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3964.218903] ? watchdog+0x40/0x40 [ 3964.220143] ? hrtimer_interrupt+0x128/0x2c0 [ 3964.224015] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3964.229662] ? apic_timer_interrupt+0xf/0x20 [ 3964.232198] [ 3964.232874] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3964.235845] do_raw_spin_lock+0x177/0x190 [ 3964.243172] _raw_spin_lock+0x21/0x30 [ 3964.248499] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3964.252140] ? do_raw_spin_unlock+0x75/0x190 [ 3964.255355] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3964.257255] ? do_wait_intr+0xf0/0xf0 [ 3964.259821] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3964.263588] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3964.265610] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3964.274476] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3964.281552] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3964.290585] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3964.292321] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3964.295229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3964.300621] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3964.306142] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3964.309096] lmv_intent_open+0x338/0x1150 [lmv] [ 3964.312720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3964.317452] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3964.319354] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3964.323544] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3964.327418] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3964.329193] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3964.345569] ? do_raw_spin_unlock+0x75/0x190 [ 3964.346967] ? _raw_spin_unlock+0x12/0x30 [ 3964.352111] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3964.354294] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3964.356561] ll_atomic_open+0x36a/0xd70 [lustre] [ 3964.360470] lookup_open+0x356/0xb40 [ 3964.371014] do_last+0x313/0xfc0 [ 3964.373723] ? nd_jump_root+0xe5/0x160 [ 3964.378246] ? path_init+0x437/0x520 [ 3964.380762] path_openat+0xf7/0x500 [ 3964.389250] ? do_raw_spin_unlock+0x75/0x190 [ 3964.390717] do_filp_open+0x99/0x140 [ 3964.391988] ? getname_flags+0x6e/0x330 [ 3964.398950] ? __check_object_size+0xff/0x256 [ 3964.401480] ? do_raw_spin_unlock+0x75/0x190 [ 3964.403809] ? _raw_spin_unlock+0x12/0x30 [ 3964.407575] do_sys_openat2+0x2b4/0x410 [ 3964.410369] do_sys_open+0x73/0xa0 [ 3964.411374] __x64_sys_openat+0x24/0x30 [ 3964.415405] do_syscall_64+0xc1/0x440 [ 3964.419181] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3964.422627] RIP: 0033:0x7fdc0afbf332 [ 3964.428642] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3964.437695] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3964.440389] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3964.443138] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3964.445838] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3964.448503] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3964.451420] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 3976.048016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 3976.050510] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3976.051899] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 3976.075305] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3976.075321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3976.075324] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3976.075339] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3976.075344] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3976.075349] RAX: 0000000000004091 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3976.075351] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 3976.075353] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 3976.075355] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 3976.075357] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 3976.075360] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 3976.075362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3976.075364] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 3976.075372] Call Trace: [ 3976.075378] [ 3976.075384] ? show_regs.cold.9+0x22/0x2f [ 3976.075394] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3976.075402] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3976.075405] ? watchdog+0x40/0x40 [ 3976.075409] ? hrtimer_interrupt+0x128/0x2c0 [ 3976.075413] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3976.075422] ? apic_timer_interrupt+0xf/0x20 [ 3976.075426] [ 3976.075428] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3976.075433] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3976.075437] ? dump_stack+0xf6/0x10e [ 3976.075442] do_raw_spin_lock+0x177/0x190 [ 3976.075447] _raw_spin_lock+0x21/0x30 [ 3976.075455] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3976.075729] obd_mod_free+0x46/0x140 [ptlrpc] [ 3976.076039] mdc_commit_open+0x91/0xb0 [mdc] [ 3976.076151] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3976.076359] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3976.076564] after_reply+0xe2c/0x1780 [ptlrpc] [ 3976.076764] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3976.077035] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3976.077375] ? do_wait_intr+0xf0/0xf0 [ 3976.077385] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3976.077584] kthread+0x1d1/0x200 [ 3976.077591] ? set_kthread_struct+0x70/0x70 [ 3976.077594] ret_from_fork+0x1f/0x30 [ 3976.216640] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3976.245550] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3976.254852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3976.258619] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3976.260342] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3976.265411] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3976.277327] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3976.279419] RAX: 0000000000005e88 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3976.286128] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 3976.287965] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3976.292408] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 3976.296474] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 3976.300180] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 3976.305074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3976.308173] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 3976.311925] Call Trace: [ 3976.315999] [ 3976.318499] ? show_regs.cold.9+0x22/0x2f [ 3976.320402] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3976.326789] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3976.329759] ? watchdog+0x40/0x40 [ 3976.332917] ? hrtimer_interrupt+0x128/0x2c0 [ 3976.340044] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3976.345838] ? apic_timer_interrupt+0xf/0x20 [ 3976.348024] [ 3976.351539] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3976.354214] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3976.357697] do_raw_spin_lock+0x177/0x190 [ 3976.358902] _raw_spin_lock+0x21/0x30 [ 3976.361241] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3976.365182] process_one_work+0x2c8/0x700 [ 3976.370168] worker_thread+0x296/0x6e0 [ 3976.371278] ? rescuer_thread+0x570/0x570 [ 3976.376754] kthread+0x1d1/0x200 [ 3976.379985] ? set_kthread_struct+0x70/0x70 [ 3976.384838] ret_from_fork+0x1f/0x30 [ 3981.791603] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1956s! [ 3981.803585] Showing busy workqueues and worker pools: [ 3981.810097] workqueue events: flags=0x0 [ 3981.815076] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3981.822044] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3981.830647] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3981.839711] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 3981.859945] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3981.862145] pending: kfree_rcu_monitor [ 3981.871118] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3981.881001] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 3981.896105] workqueue events_long: flags=0x0 [ 3981.899318] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3981.904276] pending: lu_objects_flush [obdclass] [ 3981.906213] workqueue events_unbound: flags=0x2 [ 3981.914107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3981.921615] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3981.934282] workqueue events_power_efficient: flags=0x80 [ 3981.942936] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3981.955252] pending: neigh_periodic_work, check_lifetime [ 3981.965384] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3981.979276] pending: do_cache_clean, neigh_periodic_work [ 3981.988697] workqueue mm_percpu_wq: flags=0x8 [ 3981.994787] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3981.997047] pending: lru_add_drain_per_cpu [ 3982.004711] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3982.008701] pending: vmstat_update, lru_add_drain_per_cpu [ 3982.014925] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3982.018581] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3982.026684] workqueue writeback: flags=0x4a [ 3982.031167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3982.040549] pending: wb_workfn, wb_workfn [ 3982.042404] workqueue xprtiod: flags=0xa [ 3982.043796] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3982.053548] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3982.062899] workqueue ipv6_addrconf: flags=0x40008 [ 3982.065815] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3982.072534] pending: addrconf_verify_work [ 3982.081053] workqueue knbd0-recv: flags=0x1a [ 3982.082582] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3982.089970] in-flight: 655:recv_work [ 3982.093531] workqueue ptlrpc_pinger: flags=0x2 [ 3982.097275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3982.105908] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 3982.111094] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1957s workers=4 idle: 36396 38 11188 [ 3982.125179] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3972s workers=2 idle: 658 [ 3988.056640] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 3988.065802] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3988.090875] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3988.094907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3988.105586] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3988.116355] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3988.134368] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3988.146165] RAX: 0000000000001511 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 3988.153862] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 3988.160916] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3988.165013] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 3988.169754] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 3988.173008] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 3988.177276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3988.180788] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 3988.184339] Call Trace: [ 3988.185285] [ 3988.186245] ? show_regs.cold.9+0x22/0x2f [ 3988.188686] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3988.191309] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3988.194843] ? watchdog+0x40/0x40 [ 3988.198117] ? hrtimer_interrupt+0x128/0x2c0 [ 3988.200204] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3988.205738] ? apic_timer_interrupt+0xf/0x20 [ 3988.211736] [ 3988.212435] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3988.215298] do_raw_spin_lock+0x177/0x190 [ 3988.219820] _raw_spin_lock+0x21/0x30 [ 3988.225146] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3988.231794] ? do_raw_spin_unlock+0x75/0x190 [ 3988.236842] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3988.244264] ? do_wait_intr+0xf0/0xf0 [ 3988.246349] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3988.254242] mdc_close+0x53b/0xd70 [mdc] [ 3988.256878] lmv_close+0x18d/0x490 [lmv] [ 3988.259448] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3988.262210] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3988.265395] ll_md_close+0x274/0x800 [lustre] [ 3988.267966] ll_file_release+0x128/0xc50 [lustre] [ 3988.275875] __fput+0x10d/0x440 [ 3988.277056] ____fput+0x12/0x20 [ 3988.278967] task_work_run+0xc6/0x110 [ 3988.281338] exit_to_usermode_loop+0x1dd/0x1f0 [ 3988.285134] do_syscall_64+0x3d6/0x440 [ 3988.287151] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3988.289689] RIP: 0033:0x7fdc0afbfcd8 [ 3988.292097] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3988.310946] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3988.313953] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 3988.317347] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 3988.324786] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 3988.327136] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 3988.331275] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3992.052125] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 3992.055793] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3992.077641] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3992.083224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3992.089085] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3992.091921] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3992.101087] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3992.104934] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 3992.108781] RDX: 00000000000014b9 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 3992.111737] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 3992.115181] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 3992.118251] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3992.122380] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 3992.125673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3992.127607] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 3992.130814] Call Trace: [ 3992.131932] [ 3992.132865] ? show_regs.cold.9+0x22/0x2f [ 3992.134961] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3992.137428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3992.139922] ? watchdog+0x40/0x40 [ 3992.141780] ? hrtimer_interrupt+0x128/0x2c0 [ 3992.143829] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3992.145927] ? apic_timer_interrupt+0xf/0x20 [ 3992.148442] [ 3992.149721] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3992.152656] do_raw_spin_lock+0x177/0x190 [ 3992.155018] _raw_spin_lock+0x21/0x30 [ 3992.157516] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3992.160455] ? do_raw_spin_unlock+0x75/0x190 [ 3992.163324] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3992.166230] ? do_wait_intr+0xf0/0xf0 [ 3992.168305] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3992.171902] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3992.174664] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3992.177171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3992.180131] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3992.182528] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3992.185198] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3992.186909] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3992.189517] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3992.191773] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3992.193655] lmv_intent_open+0x338/0x1150 [lmv] [ 3992.194985] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3992.196670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3992.198319] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3992.199770] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3992.201473] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3992.203186] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3992.204584] ? do_raw_spin_unlock+0x75/0x190 [ 3992.205837] ? _raw_spin_unlock+0x12/0x30 [ 3992.206981] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3992.208679] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3992.210206] ll_atomic_open+0x36a/0xd70 [lustre] [ 3992.211730] lookup_open+0x356/0xb40 [ 3992.212800] do_last+0x313/0xfc0 [ 3992.213888] ? nd_jump_root+0xe5/0x160 [ 3992.215022] ? path_init+0x437/0x520 [ 3992.216136] path_openat+0xf7/0x500 [ 3992.217117] ? do_raw_spin_unlock+0x75/0x190 [ 3992.218280] do_filp_open+0x99/0x140 [ 3992.219308] ? getname_flags+0x6e/0x330 [ 3992.220486] ? __check_object_size+0xff/0x256 [ 3992.221830] ? do_raw_spin_unlock+0x75/0x190 [ 3992.223173] ? _raw_spin_unlock+0x12/0x30 [ 3992.224486] do_sys_openat2+0x2b4/0x410 [ 3992.225732] do_sys_open+0x73/0xa0 [ 3992.226686] __x64_sys_openat+0x24/0x30 [ 3992.227933] do_syscall_64+0xc1/0x440 [ 3992.228961] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3992.230724] RIP: 0033:0x7fdc0afbf332 [ 3992.231821] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3992.238450] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3992.240926] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 3992.243632] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 3992.247244] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 3992.250984] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 3992.254166] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4004.054892] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 4004.061819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4004.093435] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4004.099305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4004.102440] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4004.106132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4004.108117] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4004.115222] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4004.118165] RAX: 0000000000005490 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4004.122534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4004.125638] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4004.128968] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4004.132219] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4004.135622] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4004.138966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4004.141586] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4004.144720] Call Trace: [ 4004.147173] [ 4004.148117] ? show_regs.cold.9+0x22/0x2f [ 4004.149988] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4004.152101] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4004.153891] ? watchdog+0x40/0x40 [ 4004.155630] ? hrtimer_interrupt+0x128/0x2c0 [ 4004.157221] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4004.160500] ? apic_timer_interrupt+0xf/0x20 [ 4004.162703] [ 4004.164062] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4004.166785] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4004.170065] do_raw_spin_lock+0x177/0x190 [ 4004.172614] _raw_spin_lock+0x21/0x30 [ 4004.176776] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4004.178937] process_one_work+0x2c8/0x700 [ 4004.180610] worker_thread+0x296/0x6e0 [ 4004.182103] ? rescuer_thread+0x570/0x570 [ 4004.183608] kthread+0x1d1/0x200 [ 4004.184710] ? set_kthread_struct+0x70/0x70 [ 4004.187671] ret_from_fork+0x1f/0x30 [ 4008.048047] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4008.053564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4008.083942] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4008.090039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4008.095510] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4008.098039] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4008.105765] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4008.108811] RAX: 0000000000003546 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4008.111122] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4008.113783] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4008.116564] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4008.119792] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4008.124254] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4008.127612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4008.129997] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4008.133191] Call Trace: [ 4008.134311] [ 4008.135219] ? show_regs.cold.9+0x22/0x2f [ 4008.136816] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4008.138751] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4008.140313] ? watchdog+0x40/0x40 [ 4008.141602] ? hrtimer_interrupt+0x128/0x2c0 [ 4008.143008] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4008.144649] ? apic_timer_interrupt+0xf/0x20 [ 4008.146618] [ 4008.147721] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4008.151004] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4008.153943] ? dump_stack+0xf6/0x10e [ 4008.156781] do_raw_spin_lock+0x177/0x190 [ 4008.158750] _raw_spin_lock+0x21/0x30 [ 4008.160620] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4008.163121] obd_mod_free+0x46/0x140 [ptlrpc] [ 4008.165791] mdc_commit_open+0x91/0xb0 [mdc] [ 4008.167756] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4008.170240] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4008.172932] after_reply+0xe2c/0x1780 [ptlrpc] [ 4008.175373] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4008.178002] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4008.180256] ? do_wait_intr+0xf0/0xf0 [ 4008.182154] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4008.184955] kthread+0x1d1/0x200 [ 4008.186719] ? set_kthread_struct+0x70/0x70 [ 4008.188700] ret_from_fork+0x1f/0x30 [ 4012.513855] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1987s! [ 4012.518277] Showing busy workqueues and worker pools: [ 4012.521557] workqueue events: flags=0x0 [ 4012.523689] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4012.532551] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4012.535996] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4012.539576] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4012.550973] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4012.559834] pending: kfree_rcu_monitor [ 4012.568544] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4012.576078] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4012.582835] workqueue events_long: flags=0x0 [ 4012.585322] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4012.589805] pending: lu_objects_flush [obdclass] [ 4012.596336] workqueue events_unbound: flags=0x2 [ 4012.600460] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4012.603570] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4012.621987] workqueue events_power_efficient: flags=0x80 [ 4012.627752] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4012.631517] pending: neigh_periodic_work, check_lifetime [ 4012.635529] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4012.640511] pending: do_cache_clean, neigh_periodic_work [ 4012.643636] workqueue mm_percpu_wq: flags=0x8 [ 4012.646721] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4012.654829] pending: lru_add_drain_per_cpu [ 4012.656857] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4012.668736] pending: vmstat_update, lru_add_drain_per_cpu [ 4012.676913] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4012.686261] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4012.690608] workqueue writeback: flags=0x4a [ 4012.696054] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4012.700637] pending: wb_workfn, wb_workfn [ 4012.704933] workqueue xprtiod: flags=0xa [ 4012.706722] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4012.712855] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4012.720333] workqueue ipv6_addrconf: flags=0x40008 [ 4012.725867] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4012.731376] pending: addrconf_verify_work [ 4012.736195] workqueue knbd0-recv: flags=0x1a [ 4012.737820] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4012.745253] in-flight: 655:recv_work [ 4012.748799] workqueue ptlrpc_pinger: flags=0x2 [ 4012.752432] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4012.757024] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4012.763650] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1987s workers=4 idle: 36396 38 11188 [ 4012.779797] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4003s workers=2 idle: 658 [ 4016.063835] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4016.067812] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4016.094970] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4016.102591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4016.106825] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4016.110465] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4016.120298] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4016.124965] RAX: 0000000000001009 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4016.129192] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4016.132745] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4016.135584] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4016.138335] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4016.140524] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4016.143467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4016.146073] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4016.149733] Call Trace: [ 4016.150762] [ 4016.151513] ? show_regs.cold.9+0x22/0x2f [ 4016.152885] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4016.154364] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4016.156073] ? watchdog+0x40/0x40 [ 4016.157628] ? hrtimer_interrupt+0x128/0x2c0 [ 4016.159988] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4016.162034] ? apic_timer_interrupt+0xf/0x20 [ 4016.164029] [ 4016.165057] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4016.167962] do_raw_spin_lock+0x177/0x190 [ 4016.169942] _raw_spin_lock+0x21/0x30 [ 4016.171700] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4016.174505] ? do_raw_spin_unlock+0x75/0x190 [ 4016.176449] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4016.179627] ? do_wait_intr+0xf0/0xf0 [ 4016.181766] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4016.184474] mdc_close+0x53b/0xd70 [mdc] [ 4016.186104] lmv_close+0x18d/0x490 [lmv] [ 4016.188037] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4016.190246] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4016.192603] ll_md_close+0x274/0x800 [lustre] [ 4016.194816] ll_file_release+0x128/0xc50 [lustre] [ 4016.197110] __fput+0x10d/0x440 [ 4016.199263] ____fput+0x12/0x20 [ 4016.201107] task_work_run+0xc6/0x110 [ 4016.202941] exit_to_usermode_loop+0x1dd/0x1f0 [ 4016.204429] do_syscall_64+0x3d6/0x440 [ 4016.206817] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4016.209115] RIP: 0033:0x7fdc0afbfcd8 [ 4016.211082] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4016.218894] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4016.222534] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4016.226745] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4016.230535] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4016.239044] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4016.244762] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4020.055516] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4020.061425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4020.125293] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4020.134895] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4020.139406] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4020.140865] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4020.170094] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4020.177148] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4020.182815] RDX: 0000000000007535 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4020.194314] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4020.200390] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4020.208495] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4020.223774] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4020.225939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4020.235250] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4020.251792] Call Trace: [ 4020.252649] [ 4020.253344] ? show_regs.cold.9+0x22/0x2f [ 4020.266423] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4020.272829] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4020.276533] ? watchdog+0x40/0x40 [ 4020.282147] ? hrtimer_interrupt+0x128/0x2c0 [ 4020.287238] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4020.299113] ? apic_timer_interrupt+0xf/0x20 [ 4020.304403] [ 4020.308202] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4020.309523] do_raw_spin_lock+0x177/0x190 [ 4020.310443] _raw_spin_lock+0x21/0x30 [ 4020.311344] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4020.326310] ? do_raw_spin_unlock+0x75/0x190 [ 4020.337862] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4020.346828] ? do_wait_intr+0xf0/0xf0 [ 4020.353103] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4020.355128] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4020.360402] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4020.369188] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4020.372544] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4020.375224] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4020.377892] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4020.380436] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4020.385267] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4020.388443] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4020.391217] lmv_intent_open+0x338/0x1150 [lmv] [ 4020.393950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4020.398690] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4020.401846] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4020.407709] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4020.412890] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4020.419504] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4020.421847] ? do_raw_spin_unlock+0x75/0x190 [ 4020.423332] ? _raw_spin_unlock+0x12/0x30 [ 4020.433622] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4020.439819] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4020.444436] ll_atomic_open+0x36a/0xd70 [lustre] [ 4020.446142] lookup_open+0x356/0xb40 [ 4020.455758] do_last+0x313/0xfc0 [ 4020.456967] ? nd_jump_root+0xe5/0x160 [ 4020.463332] ? path_init+0x437/0x520 [ 4020.468740] path_openat+0xf7/0x500 [ 4020.473391] ? do_raw_spin_unlock+0x75/0x190 [ 4020.476545] do_filp_open+0x99/0x140 [ 4020.481370] ? getname_flags+0x6e/0x330 [ 4020.485130] ? __check_object_size+0xff/0x256 [ 4020.489494] ? do_raw_spin_unlock+0x75/0x190 [ 4020.495231] ? _raw_spin_unlock+0x12/0x30 [ 4020.499542] do_sys_openat2+0x2b4/0x410 [ 4020.504519] do_sys_open+0x73/0xa0 [ 4020.506422] __x64_sys_openat+0x24/0x30 [ 4020.507802] do_syscall_64+0xc1/0x440 [ 4020.509060] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4020.514424] RIP: 0033:0x7fdc0afbf332 [ 4020.515717] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4020.540539] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4020.547714] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4020.554482] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4020.564835] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4020.572565] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4020.576873] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4032.051111] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 4032.058728] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4032.085652] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4032.091103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4032.095109] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4032.098397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4032.100677] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4032.107171] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4032.109556] RAX: 0000000000007f10 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4032.112346] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4032.114899] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4032.118001] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4032.120958] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4032.124237] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4032.127244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4032.128961] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4032.131039] Call Trace: [ 4032.131759] [ 4032.132233] ? show_regs.cold.9+0x22/0x2f [ 4032.133544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4032.134987] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4032.136722] ? watchdog+0x40/0x40 [ 4032.137801] ? hrtimer_interrupt+0x128/0x2c0 [ 4032.139570] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4032.141517] ? apic_timer_interrupt+0xf/0x20 [ 4032.142935] [ 4032.143574] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4032.145188] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4032.147155] do_raw_spin_lock+0x177/0x190 [ 4032.148846] _raw_spin_lock+0x21/0x30 [ 4032.149917] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4032.151935] process_one_work+0x2c8/0x700 [ 4032.153187] worker_thread+0x296/0x6e0 [ 4032.154338] ? rescuer_thread+0x570/0x570 [ 4032.155719] kthread+0x1d1/0x200 [ 4032.156732] ? set_kthread_struct+0x70/0x70 [ 4032.157984] ret_from_fork+0x1f/0x30 [ 4040.048508] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4040.053542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4040.105907] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4040.113395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4040.116561] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4040.117931] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4040.132553] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4040.139306] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4040.143750] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4040.153600] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4040.157938] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4040.160676] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4040.170544] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4040.175506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4040.178451] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4040.183532] Call Trace: [ 4040.187411] [ 4040.191523] ? show_regs.cold.9+0x22/0x2f [ 4040.192918] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4040.200806] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4040.203546] ? watchdog+0x40/0x40 [ 4040.206175] ? hrtimer_interrupt+0x128/0x2c0 [ 4040.207976] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4040.210609] ? apic_timer_interrupt+0xf/0x20 [ 4040.213818] [ 4040.215020] ? native_safe_halt+0x2e/0x50 [ 4040.218017] kvm_wait+0xcd/0xe0 [ 4040.219768] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4040.223306] ? dump_stack+0xf6/0x10e [ 4040.232279] do_raw_spin_lock+0x177/0x190 [ 4040.234265] _raw_spin_lock+0x21/0x30 [ 4040.237445] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4040.241147] obd_mod_free+0x46/0x140 [ptlrpc] [ 4040.243847] mdc_commit_open+0x91/0xb0 [mdc] [ 4040.250321] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4040.257798] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4040.262354] after_reply+0xe2c/0x1780 [ptlrpc] [ 4040.264771] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4040.266853] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4040.273044] ? do_wait_intr+0xf0/0xf0 [ 4040.276601] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4040.288256] kthread+0x1d1/0x200 [ 4040.290251] ? set_kthread_struct+0x70/0x70 [ 4040.292686] ret_from_fork+0x1f/0x30 [ 4043.231456] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2018s! [ 4043.245350] Showing busy workqueues and worker pools: [ 4043.247598] workqueue events: flags=0x0 [ 4043.255346] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4043.258425] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4043.263063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4043.267034] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4043.282261] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4043.290291] pending: kfree_rcu_monitor [ 4043.296931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4043.305816] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4043.317304] workqueue events_long: flags=0x0 [ 4043.321249] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4043.328352] pending: lu_objects_flush [obdclass] [ 4043.337784] workqueue events_unbound: flags=0x2 [ 4043.347931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4043.354038] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4043.359254] workqueue events_power_efficient: flags=0x80 [ 4043.361289] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4043.369979] pending: neigh_periodic_work, check_lifetime [ 4043.375542] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4043.386623] pending: do_cache_clean, neigh_periodic_work [ 4043.391975] workqueue mm_percpu_wq: flags=0x8 [ 4043.396708] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4043.403677] pending: lru_add_drain_per_cpu [ 4043.407408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4043.412666] pending: vmstat_update, lru_add_drain_per_cpu [ 4043.416502] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4043.422342] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4043.429660] workqueue writeback: flags=0x4a [ 4043.431172] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4043.434541] pending: wb_workfn, wb_workfn [ 4043.440577] workqueue xprtiod: flags=0xa [ 4043.444942] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4043.447988] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4043.453286] workqueue ipv6_addrconf: flags=0x40008 [ 4043.455014] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4043.458353] pending: addrconf_verify_work [ 4043.460963] workqueue knbd0-recv: flags=0x1a [ 4043.465228] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4043.481627] in-flight: 655:recv_work [ 4043.489812] workqueue ptlrpc_pinger: flags=0x2 [ 4043.491496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4043.502313] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4043.507068] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2018s workers=4 idle: 36396 38 11188 [ 4043.514811] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4034s workers=2 idle: 658 [ 4044.054096] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4044.057750] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4044.080277] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4044.085415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4044.089591] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4044.092426] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4044.102310] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4044.107417] RAX: 0000000000002683 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4044.112161] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4044.116245] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4044.120320] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4044.123360] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4044.126055] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4044.129581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4044.131624] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4044.134609] Call Trace: [ 4044.135499] [ 4044.136226] ? show_regs.cold.9+0x22/0x2f [ 4044.138448] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4044.140529] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4044.143006] ? watchdog+0x40/0x40 [ 4044.144504] ? hrtimer_interrupt+0x128/0x2c0 [ 4044.146465] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4044.150091] ? apic_timer_interrupt+0xf/0x20 [ 4044.152237] [ 4044.153426] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4044.155887] do_raw_spin_lock+0x177/0x190 [ 4044.157756] _raw_spin_lock+0x21/0x30 [ 4044.159493] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4044.162358] ? do_raw_spin_unlock+0x75/0x190 [ 4044.163809] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4044.165920] ? do_wait_intr+0xf0/0xf0 [ 4044.167325] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4044.169186] mdc_close+0x53b/0xd70 [mdc] [ 4044.170613] lmv_close+0x18d/0x490 [lmv] [ 4044.172366] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4044.174249] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4044.176145] ll_md_close+0x274/0x800 [lustre] [ 4044.177791] ll_file_release+0x128/0xc50 [lustre] [ 4044.179968] __fput+0x10d/0x440 [ 4044.180834] ____fput+0x12/0x20 [ 4044.181987] task_work_run+0xc6/0x110 [ 4044.183450] exit_to_usermode_loop+0x1dd/0x1f0 [ 4044.184725] do_syscall_64+0x3d6/0x440 [ 4044.185952] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4044.187610] RIP: 0033:0x7fdc0afbfcd8 [ 4044.188966] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4044.195742] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4044.198347] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4044.201011] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4044.203545] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4044.206086] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4044.208438] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4048.056113] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 4048.060686] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4048.091079] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4048.098492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4048.104701] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4048.107221] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4048.120470] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4048.124017] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4048.127168] RDX: 000000000000355c RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4048.130321] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4048.133585] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4048.136773] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4048.139875] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4048.143730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4048.145846] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4048.148896] Call Trace: [ 4048.150053] [ 4048.150878] ? show_regs.cold.9+0x22/0x2f [ 4048.152869] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4048.154959] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4048.157507] ? watchdog+0x40/0x40 [ 4048.159046] ? hrtimer_interrupt+0x128/0x2c0 [ 4048.160893] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4048.167162] ? apic_timer_interrupt+0xf/0x20 [ 4048.169544] [ 4048.170308] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4048.172473] do_raw_spin_lock+0x177/0x190 [ 4048.174214] _raw_spin_lock+0x21/0x30 [ 4048.176082] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4048.178406] ? do_raw_spin_unlock+0x75/0x190 [ 4048.180695] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4048.183245] ? do_wait_intr+0xf0/0xf0 [ 4048.184779] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4048.187322] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4048.189913] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4048.192389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.195083] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4048.197460] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4048.199225] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4048.201424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.204060] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4048.206900] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4048.209545] lmv_intent_open+0x338/0x1150 [lmv] [ 4048.211379] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.214180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.218958] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4048.220959] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4048.225106] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.227455] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4048.229713] ? do_raw_spin_unlock+0x75/0x190 [ 4048.231163] ? _raw_spin_unlock+0x12/0x30 [ 4048.233507] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4048.238310] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4048.241670] ll_atomic_open+0x36a/0xd70 [lustre] [ 4048.244726] lookup_open+0x356/0xb40 [ 4048.247493] do_last+0x313/0xfc0 [ 4048.249207] ? nd_jump_root+0xe5/0x160 [ 4048.250355] ? path_init+0x437/0x520 [ 4048.251785] path_openat+0xf7/0x500 [ 4048.252979] ? do_raw_spin_unlock+0x75/0x190 [ 4048.254452] do_filp_open+0x99/0x140 [ 4048.255994] ? getname_flags+0x6e/0x330 [ 4048.257406] ? __check_object_size+0xff/0x256 [ 4048.259481] ? do_raw_spin_unlock+0x75/0x190 [ 4048.261293] ? _raw_spin_unlock+0x12/0x30 [ 4048.265888] do_sys_openat2+0x2b4/0x410 [ 4048.269199] do_sys_open+0x73/0xa0 [ 4048.271035] __x64_sys_openat+0x24/0x30 [ 4048.273163] do_syscall_64+0xc1/0x440 [ 4048.276271] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4048.278147] RIP: 0033:0x7fdc0afbf332 [ 4048.280107] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4048.291606] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4048.294758] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4048.297478] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4048.299753] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4048.303159] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4048.306234] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4059.388022] rcu: INFO: rcu_sched self-detected stall on CPU [ 4059.390359] rcu: 0-...!: (1186990 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 4059.394433] (t=2040045 jiffies g=270761 q=3200) [ 4059.396018] rcu: rcu_sched kthread starved for 2034977 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 4059.400458] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4059.404030] rcu: RCU grace-period kthread stack dump: [ 4059.406333] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4059.410326] Call Trace: [ 4059.411184] __schedule+0x351/0xcb0 [ 4059.412568] schedule+0xc0/0x180 [ 4059.413884] schedule_timeout+0xb4/0x190 [ 4059.415644] ? do_raw_spin_unlock+0x75/0x190 [ 4059.417684] ? __next_timer_interrupt+0x160/0x160 [ 4059.419694] ? prepare_to_swait_event+0xf3/0x140 [ 4059.421623] rcu_gp_kthread+0x8dd/0x15b0 [ 4059.423198] ? force_qs_rnp+0x2c0/0x2c0 [ 4059.424765] kthread+0x1d1/0x200 [ 4059.426054] ? set_kthread_struct+0x70/0x70 [ 4059.427586] ret_from_fork+0x1f/0x30 [ 4059.428848] rcu: Stack dump where RCU GP kthread last ran: [ 4059.430845] Sending NMI from CPU 0 to CPUs 3: [ 4059.433640] NMI backtrace for cpu 3 [ 4059.433646] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4059.433648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4059.433650] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4059.433651] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4059.433653] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 [ 4059.433656] RAX: 0000000000006687 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4059.433657] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4059.433659] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4059.433660] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4059.433661] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4059.433663] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4059.433664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4059.433665] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4059.433666] Call Trace: [ 4059.433667] [ 4059.433668] ? show_regs.cold.9+0x22/0x2f [ 4059.433669] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4059.433670] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4059.433671] ? nmi_handle+0xbd/0x210 [ 4059.433672] ? default_do_nmi+0x4b/0x1b0 [ 4059.433673] ? do_nmi+0x280/0x380 [ 4059.433674] ? end_repeat_nmi+0x16/0x1f [ 4059.433675] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4059.433677] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4059.433678] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4059.433679] [ 4059.433680] do_raw_spin_lock+0x177/0x190 [ 4059.433681] _raw_spin_lock+0x21/0x30 [ 4059.433682] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4059.433684] ? do_raw_spin_unlock+0x75/0x190 [ 4059.433685] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4059.433686] ? do_wait_intr+0xf0/0xf0 [ 4059.433687] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4059.433688] mdc_close+0x53b/0xd70 [mdc] [ 4059.433690] lmv_close+0x18d/0x490 [lmv] [ 4059.433691] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4059.433692] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4059.433693] ll_md_close+0x274/0x800 [lustre] [ 4059.433694] ll_file_release+0x128/0xc50 [lustre] [ 4059.433696] __fput+0x10d/0x440 [ 4059.433697] ____fput+0x12/0x20 [ 4059.433698] task_work_run+0xc6/0x110 [ 4059.433699] exit_to_usermode_loop+0x1dd/0x1f0 [ 4059.433700] do_syscall_64+0x3d6/0x440 [ 4059.433701] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4059.433703] RIP: 0033:0x7fdc0afbfcd8 [ 4059.433704] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4059.433706] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4059.433708] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4059.433710] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4059.433711] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4059.433712] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4059.433713] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4059.434509] NMI backtrace for cpu 0 [ 4059.584029] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4059.588981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4059.592993] Call Trace: [ 4059.594019] [ 4059.594825] ? dump_stack+0xbb/0x10e [ 4059.596199] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4059.598104] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4059.600026] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4059.602068] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4059.604488] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4059.606506] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4059.608581] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4059.610257] ? update_process_times+0x6d/0xc0 [ 4059.611745] ? tick_sched_handle.isra.18+0x27/0x90 [ 4059.613468] ? tick_sched_timer+0x53/0xb0 [ 4059.614998] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4059.616553] ? tick_sched_handle.isra.18+0x90/0x90 [ 4059.618277] ? hrtimer_interrupt+0x128/0x2c0 [ 4059.619568] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4059.621248] ? apic_timer_interrupt+0xf/0x20 [ 4059.622610] [ 4059.623211] ? native_safe_halt+0x2e/0x50 [ 4059.624441] ? kvm_wait+0xcd/0xe0 [ 4059.625485] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4059.627310] ? dump_stack+0xf6/0x10e [ 4059.628492] ? do_raw_spin_lock+0x177/0x190 [ 4059.629539] ? _raw_spin_lock+0x21/0x30 [ 4059.630793] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4059.632993] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4059.634748] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4059.636107] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4059.638528] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4059.640571] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4059.642417] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4059.644521] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4059.646161] ? do_wait_intr+0xf0/0xf0 [ 4059.647335] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4059.649661] ? kthread+0x1d1/0x200 [ 4059.650572] ? set_kthread_struct+0x70/0x70 [ 4059.651699] ? ret_from_fork+0x1f/0x30 [ 4060.051031] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 4060.053620] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4060.096026] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4060.101174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4060.110335] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4060.116155] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4060.120583] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4060.137153] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4060.145115] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4060.153116] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4060.157446] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4060.164943] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4060.170650] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4060.177626] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4060.184649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4060.189365] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4060.196325] Call Trace: [ 4060.197021] [ 4060.197586] ? show_regs.cold.9+0x22/0x2f [ 4060.202246] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4060.207779] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4060.210665] ? watchdog+0x40/0x40 [ 4060.212103] ? hrtimer_interrupt+0x128/0x2c0 [ 4060.215222] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4060.219096] ? apic_timer_interrupt+0xf/0x20 [ 4060.222870] [ 4060.224857] ? native_safe_halt+0x2e/0x50 [ 4060.226641] kvm_wait+0xcd/0xe0 [ 4060.228038] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4060.230622] do_raw_spin_lock+0x177/0x190 [ 4060.232581] _raw_spin_lock+0x21/0x30 [ 4060.234151] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4060.236203] process_one_work+0x2c8/0x700 [ 4060.238113] worker_thread+0x296/0x6e0 [ 4060.239702] ? rescuer_thread+0x570/0x570 [ 4060.242129] kthread+0x1d1/0x200 [ 4060.243584] ? set_kthread_struct+0x70/0x70 [ 4060.245466] ret_from_fork+0x1f/0x30 [ 4073.953785] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2049s! [ 4073.969081] Showing busy workqueues and worker pools: [ 4073.976781] workqueue events: flags=0x0 [ 4073.981578] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4073.987058] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4073.991474] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4073.995929] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4074.003805] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4074.007894] pending: kfree_rcu_monitor [ 4074.009528] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4074.013534] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4074.020895] workqueue events_long: flags=0x0 [ 4074.023019] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4074.026397] pending: lu_objects_flush [obdclass] [ 4074.028316] workqueue events_unbound: flags=0x2 [ 4074.030505] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4074.033503] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4074.038830] workqueue events_power_efficient: flags=0x80 [ 4074.041283] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4074.044076] pending: neigh_periodic_work, check_lifetime [ 4074.046707] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4074.049314] pending: do_cache_clean, neigh_periodic_work [ 4074.052076] workqueue mm_percpu_wq: flags=0x8 [ 4074.054336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4074.057453] pending: lru_add_drain_per_cpu [ 4074.059080] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4074.061987] pending: vmstat_update, lru_add_drain_per_cpu [ 4074.064903] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4074.068065] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4074.070986] workqueue writeback: flags=0x4a [ 4074.072845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4074.075817] pending: wb_workfn, wb_workfn [ 4074.077741] workqueue xprtiod: flags=0xa [ 4074.079601] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4074.082672] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4074.086297] workqueue ipv6_addrconf: flags=0x40008 [ 4074.088643] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4074.091641] pending: addrconf_verify_work [ 4074.093706] workqueue knbd0-recv: flags=0x1a [ 4074.095619] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4074.098523] in-flight: 655:recv_work [ 4074.100531] workqueue ptlrpc_pinger: flags=0x2 [ 4074.104014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4074.110453] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4074.113836] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2049s workers=4 idle: 36396 38 11188 [ 4074.117701] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4064s workers=2 idle: 658 [ 4076.052130] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 4076.056263] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4076.074894] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4076.079352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4076.082782] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4076.084282] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4076.091826] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4076.094948] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4076.098228] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4076.100940] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4076.103878] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4076.106831] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4076.109832] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4076.113453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4076.115673] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4076.118589] Call Trace: [ 4076.119695] [ 4076.120372] ? show_regs.cold.9+0x22/0x2f [ 4076.121669] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4076.123624] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4076.125600] ? watchdog+0x40/0x40 [ 4076.126951] ? hrtimer_interrupt+0x128/0x2c0 [ 4076.128698] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4076.134345] ? apic_timer_interrupt+0xf/0x20 [ 4076.136547] [ 4076.137651] ? native_safe_halt+0x2e/0x50 [ 4076.139924] kvm_wait+0xcd/0xe0 [ 4076.141572] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4076.144374] do_raw_spin_lock+0x177/0x190 [ 4076.145994] _raw_spin_lock+0x21/0x30 [ 4076.150746] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4076.155486] ? do_raw_spin_unlock+0x75/0x190 [ 4076.158863] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4076.161244] ? do_wait_intr+0xf0/0xf0 [ 4076.165300] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4076.168982] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4076.173702] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4076.178888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4076.180704] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4076.186974] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4076.189410] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4076.191684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4076.194950] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4076.197919] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4076.200956] lmv_intent_open+0x338/0x1150 [lmv] [ 4076.202944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4076.205675] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4076.208515] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4076.210439] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4076.212581] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4076.215274] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4076.217407] ? do_raw_spin_unlock+0x75/0x190 [ 4076.219660] ? _raw_spin_unlock+0x12/0x30 [ 4076.221410] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4076.223624] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4076.225777] ll_atomic_open+0x36a/0xd70 [lustre] [ 4076.229886] lookup_open+0x356/0xb40 [ 4076.231548] do_last+0x313/0xfc0 [ 4076.232878] ? nd_jump_root+0xe5/0x160 [ 4076.234690] ? path_init+0x437/0x520 [ 4076.236341] path_openat+0xf7/0x500 [ 4076.237829] ? do_raw_spin_unlock+0x75/0x190 [ 4076.239672] do_filp_open+0x99/0x140 [ 4076.241493] ? getname_flags+0x6e/0x330 [ 4076.243016] ? __check_object_size+0xff/0x256 [ 4076.245328] ? do_raw_spin_unlock+0x75/0x190 [ 4076.249548] ? _raw_spin_unlock+0x12/0x30 [ 4076.253166] do_sys_openat2+0x2b4/0x410 [ 4076.258319] do_sys_open+0x73/0xa0 [ 4076.261597] __x64_sys_openat+0x24/0x30 [ 4076.264889] do_syscall_64+0xc1/0x440 [ 4076.268882] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4076.275046] RIP: 0033:0x7fdc0afbf332 [ 4076.277297] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4076.286885] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4076.290673] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4076.294283] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4076.302537] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4076.315660] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4076.321637] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4084.054218] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4084.058735] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4084.101401] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4084.110333] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4084.116740] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4084.118522] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4084.131743] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4084.134357] RAX: 0000000000007e92 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4084.137161] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4084.140147] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4084.142893] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4084.145800] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4084.148740] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4084.152000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4084.154741] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4084.157744] Call Trace: [ 4084.158724] [ 4084.159555] ? show_regs.cold.9+0x22/0x2f [ 4084.161130] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4084.162726] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4084.165583] ? watchdog+0x40/0x40 [ 4084.171003] ? hrtimer_interrupt+0x128/0x2c0 [ 4084.177745] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4084.179401] ? apic_timer_interrupt+0xf/0x20 [ 4084.191659] [ 4084.192519] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4084.196731] do_raw_spin_lock+0x177/0x190 [ 4084.202573] _raw_spin_lock+0x21/0x30 [ 4084.204371] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4084.208454] ? do_raw_spin_unlock+0x75/0x190 [ 4084.212592] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4084.214368] ? do_wait_intr+0xf0/0xf0 [ 4084.221635] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4084.226639] mdc_close+0x53b/0xd70 [mdc] [ 4084.229709] lmv_close+0x18d/0x490 [lmv] [ 4084.231258] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4084.233265] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4084.236371] ll_md_close+0x274/0x800 [lustre] [ 4084.240540] ll_file_release+0x128/0xc50 [lustre] [ 4084.244503] __fput+0x10d/0x440 [ 4084.246543] ____fput+0x12/0x20 [ 4084.247360] task_work_run+0xc6/0x110 [ 4084.248391] exit_to_usermode_loop+0x1dd/0x1f0 [ 4084.253589] do_syscall_64+0x3d6/0x440 [ 4084.254866] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4084.256629] RIP: 0033:0x7fdc0afbfcd8 [ 4084.258883] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4084.267262] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4084.270482] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4084.275437] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4084.279303] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4084.285470] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4084.288867] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4088.052974] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 4088.068540] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4088.107389] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4088.121392] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4088.131930] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4088.141395] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4088.149550] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4088.171750] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4088.183465] RAX: 000000000000738b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4088.190841] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4088.194173] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4088.196359] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4088.199351] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4088.202334] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4088.208928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4088.215673] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4088.220367] Call Trace: [ 4088.221178] [ 4088.222980] ? show_regs.cold.9+0x22/0x2f [ 4088.225340] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4088.228648] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4088.236233] ? watchdog+0x40/0x40 [ 4088.239429] ? hrtimer_interrupt+0x128/0x2c0 [ 4088.241370] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4088.246261] ? apic_timer_interrupt+0xf/0x20 [ 4088.247880] [ 4088.248664] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4088.251272] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4088.253673] do_raw_spin_lock+0x177/0x190 [ 4088.255518] _raw_spin_lock+0x21/0x30 [ 4088.257032] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4088.263130] process_one_work+0x2c8/0x700 [ 4088.264570] worker_thread+0x296/0x6e0 [ 4088.266048] ? rescuer_thread+0x570/0x570 [ 4088.267136] kthread+0x1d1/0x200 [ 4088.271761] ? set_kthread_struct+0x70/0x70 [ 4088.273085] ret_from_fork+0x1f/0x30 [ 4100.048081] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4100.051072] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4100.080081] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4100.086030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4100.090226] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4100.093150] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4100.105535] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4100.114054] RAX: 0000000000001072 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4100.119823] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4100.124893] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4100.130121] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4100.135359] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4100.138439] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4100.143627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4100.149798] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4100.155132] Call Trace: [ 4100.155977] [ 4100.157524] ? show_regs.cold.9+0x22/0x2f [ 4100.159837] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4100.162499] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4100.164798] ? watchdog+0x40/0x40 [ 4100.166466] ? hrtimer_interrupt+0x128/0x2c0 [ 4100.168861] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4100.173875] ? apic_timer_interrupt+0xf/0x20 [ 4100.175558] [ 4100.177090] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4100.180872] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4100.185598] ? dump_stack+0xf6/0x10e [ 4100.187073] do_raw_spin_lock+0x177/0x190 [ 4100.190536] _raw_spin_lock+0x21/0x30 [ 4100.192195] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4100.196840] obd_mod_free+0x46/0x140 [ptlrpc] [ 4100.200824] mdc_commit_open+0x91/0xb0 [mdc] [ 4100.202742] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4100.206901] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4100.212870] after_reply+0xe2c/0x1780 [ptlrpc] [ 4100.217592] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4100.222064] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4100.226581] ? do_wait_intr+0xf0/0xf0 [ 4100.228388] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4100.231516] kthread+0x1d1/0x200 [ 4100.232597] ? set_kthread_struct+0x70/0x70 [ 4100.237978] ret_from_fork+0x1f/0x30 [ 4104.054595] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 4104.067401] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4104.108076] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4104.117466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4104.128655] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4104.138756] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4104.145711] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4104.149060] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4104.152556] RDX: 0000000000005561 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4104.159054] BUG: workqueue lockup - pool [ 4104.159733] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4104.159739] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4104.159741] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4104.159744] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4104.159747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4104.159750] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4104.159757] Call Trace: [ 4104.159763] [ 4104.159768] ? show_regs.cold.9+0x22/0x2f [ 4104.159779] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4104.159785] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4104.159789] ? watchdog+0x40/0x40 [ 4104.159793] ? hrtimer_interrupt+0x128/0x2c0 [ 4104.159797] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4104.159805] ? apic_timer_interrupt+0xf/0x20 [ 4104.159926] [ 4104.159929] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4104.159936] do_raw_spin_lock+0x177/0x190 [ 4104.159940] _raw_spin_lock+0x21/0x30 [ 4104.159947] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4104.160322] ? do_raw_spin_unlock+0x75/0x190 [ 4104.160327] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4104.160520] ? do_wait_intr+0xf0/0xf0 [ 4104.160528] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4104.163066] cpus=0-3 [ 4104.170468] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4104.176093] flags=0x4 nice=0 [ 4104.183769] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4104.200715] stuck for 2079s! [ 4104.200728] Showing busy workqueues and worker pools: [ 4104.200730] workqueue events: flags=0x0 [ 4104.200735] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4104.200749] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4104.201413] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4104.201424] pending: cpu_latency_work [ptlrpc] [ 4104.203907] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4104.219846] , osc_grant_work_handler [osc] [ 4104.223346] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4104.224034] , key_garbage_collector [ 4104.224076] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4104.224087] pending: kfree_rcu_monitor [ 4104.224102] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4104.224108] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4104.224951] workqueue events_long: flags=0x0 [ 4104.224953] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4104.224959] pending: lu_objects_flush [obdclass] [ 4104.225416] workqueue events_unbound: flags=0x2 [ 4104.225420] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4104.225429] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4104.225452] workqueue events_power_efficient: flags=0x80 [ 4104.225454] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4104.225461] pending: neigh_periodic_work, check_lifetime [ 4104.225474] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4104.225480] pending: do_cache_clean, neigh_periodic_work [ 4104.225492] workqueue mm_percpu_wq: flags=0x8 [ 4104.225494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4104.225500] pending: lru_add_drain_per_cpu [ 4104.225508] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4104.225515] pending: vmstat_update, lru_add_drain_per_cpu [ 4104.225523] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4104.225528] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4104.225539] workqueue writeback: flags=0x4a [ 4104.225541] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4104.225546] pending: wb_workfn, wb_workfn [ 4104.225560] workqueue xprtiod: flags=0xa [ 4104.225562] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4104.225567] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4104.225713] workqueue ipv6_addrconf: flags=0x40008 [ 4104.225716] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4104.225722] pending: addrconf_verify_work [ 4104.225734] workqueue knbd0-recv: flags=0x1a [ 4104.225736] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4104.225742] in-flight: 655:recv_work [ 4104.225751] workqueue ptlrpc_pinger: flags=0x2 [ 4104.225753] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4104.225758] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4104.226032] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2079s workers=4 idle: 36396 38 11188 [ 4104.226046] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4094s workers=2 idle: 658 [ 4104.506963] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4104.509541] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4104.512509] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4104.518313] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4104.522857] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4104.527704] lmv_intent_open+0x338/0x1150 [lmv] [ 4104.531687] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4104.535146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4104.539890] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4104.543344] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4104.548017] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4104.553422] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4104.557778] ? do_raw_spin_unlock+0x75/0x190 [ 4104.558865] ? _raw_spin_unlock+0x12/0x30 [ 4104.559863] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4104.566876] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4104.569940] ll_atomic_open+0x36a/0xd70 [lustre] [ 4104.572988] lookup_open+0x356/0xb40 [ 4104.574883] do_last+0x313/0xfc0 [ 4104.577412] ? nd_jump_root+0xe5/0x160 [ 4104.579536] ? path_init+0x437/0x520 [ 4104.585839] path_openat+0xf7/0x500 [ 4104.590046] ? do_raw_spin_unlock+0x75/0x190 [ 4104.593995] do_filp_open+0x99/0x140 [ 4104.595680] ? getname_flags+0x6e/0x330 [ 4104.600272] ? __check_object_size+0xff/0x256 [ 4104.601633] ? do_raw_spin_unlock+0x75/0x190 [ 4104.602932] ? _raw_spin_unlock+0x12/0x30 [ 4104.609441] do_sys_openat2+0x2b4/0x410 [ 4104.616075] do_sys_open+0x73/0xa0 [ 4104.618325] __x64_sys_openat+0x24/0x30 [ 4104.619971] do_syscall_64+0xc1/0x440 [ 4104.627935] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4104.635908] RIP: 0033:0x7fdc0afbf332 [ 4104.638967] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4104.649121] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4104.653913] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4104.660290] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4104.665486] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4104.670025] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4104.675000] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4112.054544] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4112.071074] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4112.183889] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4112.209552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4112.225010] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4112.234837] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4112.274375] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4112.276984] RAX: 0000000000007d20 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4112.291983] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4112.299533] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4112.304572] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4112.313821] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4112.326750] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4112.336208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4112.340907] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4112.354165] Call Trace: [ 4112.358232] [ 4112.359562] ? show_regs.cold.9+0x22/0x2f [ 4112.361901] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4112.374707] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4112.380096] ? watchdog+0x40/0x40 [ 4112.385305] ? hrtimer_interrupt+0x128/0x2c0 [ 4112.386705] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4112.401833] ? apic_timer_interrupt+0xf/0x20 [ 4112.407764] [ 4112.409285] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4112.427266] do_raw_spin_lock+0x177/0x190 [ 4112.428749] _raw_spin_lock+0x21/0x30 [ 4112.445874] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4112.457417] ? do_raw_spin_unlock+0x75/0x190 [ 4112.461622] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4112.474649] ? do_wait_intr+0xf0/0xf0 [ 4112.481131] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4112.483220] mdc_close+0x53b/0xd70 [mdc] [ 4112.495651] lmv_close+0x18d/0x490 [lmv] [ 4112.502001] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4112.510659] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4112.512481] ll_md_close+0x274/0x800 [lustre] [ 4112.515390] ll_file_release+0x128/0xc50 [lustre] [ 4112.522105] __fput+0x10d/0x440 [ 4112.523075] ____fput+0x12/0x20 [ 4112.524151] task_work_run+0xc6/0x110 [ 4112.533769] exit_to_usermode_loop+0x1dd/0x1f0 [ 4112.536223] do_syscall_64+0x3d6/0x440 [ 4112.538819] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4112.544078] RIP: 0033:0x7fdc0afbfcd8 [ 4112.551256] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4112.590738] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4112.603969] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4112.606500] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4112.617671] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4112.632065] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4112.653081] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4116.051031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4116.055117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4116.108834] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4116.122937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4116.128277] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4116.131981] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4116.133872] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4116.150203] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4116.161641] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4116.166618] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4116.174638] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4116.179244] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4116.187727] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4116.195142] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4116.199910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4116.206208] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4116.210013] Call Trace: [ 4116.213621] [ 4116.215761] ? show_regs.cold.9+0x22/0x2f [ 4116.217795] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4116.221028] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4116.223252] ? watchdog+0x40/0x40 [ 4116.224910] ? hrtimer_interrupt+0x128/0x2c0 [ 4116.227404] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4116.234331] ? apic_timer_interrupt+0xf/0x20 [ 4116.237447] [ 4116.238863] ? native_safe_halt+0x2e/0x50 [ 4116.241772] kvm_wait+0xcd/0xe0 [ 4116.244133] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4116.247261] do_raw_spin_lock+0x177/0x190 [ 4116.249073] _raw_spin_lock+0x21/0x30 [ 4116.257254] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4116.261105] process_one_work+0x2c8/0x700 [ 4116.263521] worker_thread+0x296/0x6e0 [ 4116.265896] ? rescuer_thread+0x570/0x570 [ 4116.275462] kthread+0x1d1/0x200 [ 4116.278862] ? set_kthread_struct+0x70/0x70 [ 4116.284317] ret_from_fork+0x1f/0x30 [ 4132.052265] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 4132.060494] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 4132.065468] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4132.070472] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4132.070480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4132.070483] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4132.070511] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4132.070516] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4132.070520] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4132.070522] RDX: 000000000000561f RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4132.070525] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4132.070527] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4132.070529] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4132.070532] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4132.070534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4132.070536] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4132.070545] Call Trace: [ 4132.070550] [ 4132.070555] ? show_regs.cold.9+0x22/0x2f [ 4132.071705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4132.071712] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4132.071716] ? watchdog+0x40/0x40 [ 4132.071721] ? hrtimer_interrupt+0x128/0x2c0 [ 4132.071725] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4132.071733] ? apic_timer_interrupt+0xf/0x20 [ 4132.071738] [ 4132.071742] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4132.071748] do_raw_spin_lock+0x177/0x190 [ 4132.071752] _raw_spin_lock+0x21/0x30 [ 4132.071758] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4132.073800] ? do_raw_spin_unlock+0x75/0x190 [ 4132.073809] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4132.076724] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4132.076796] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4132.076803] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4132.076806] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4132.076817] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4132.076821] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4132.076825] RAX: 0000000000006b09 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4132.076827] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4132.076829] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4132.076831] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4132.076833] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4132.076836] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4132.076839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4132.076842] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4132.076851] Call Trace: [ 4132.076855] [ 4132.076860] ? show_regs.cold.9+0x22/0x2f [ 4132.076869] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4132.076876] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4132.076879] ? watchdog+0x40/0x40 [ 4132.076883] ? hrtimer_interrupt+0x128/0x2c0 [ 4132.076889] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4132.076898] ? apic_timer_interrupt+0xf/0x20 [ 4132.076902] [ 4132.076906] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4132.076910] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4132.076914] ? dump_stack+0xf6/0x10e [ 4132.076919] do_raw_spin_lock+0x177/0x190 [ 4132.076924] _raw_spin_lock+0x21/0x30 [ 4132.076929] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4132.077181] obd_mod_free+0x46/0x140 [ptlrpc] [ 4132.077379] mdc_commit_open+0x91/0xb0 [mdc] [ 4132.077415] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4132.201934] ? do_wait_intr+0xf0/0xf0 [ 4132.201958] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4132.202252] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4132.202461] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4132.208413] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4132.208785] after_reply+0xe2c/0x1780 [ptlrpc] [ 4132.208984] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4132.209192] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4132.209393] ? do_wait_intr+0xf0/0xf0 [ 4132.209405] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4132.214269] kthread+0x1d1/0x200 [ 4132.214278] ? set_kthread_struct+0x70/0x70 [ 4132.214283] ret_from_fork+0x1f/0x30 [ 4132.684017] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4132.688675] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4132.697018] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4132.707701] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4132.719567] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4132.729034] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4132.738878] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4132.744314] lmv_intent_open+0x338/0x1150 [lmv] [ 4132.748714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4132.768616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4132.772522] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4132.778502] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4132.788806] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4132.790717] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4132.805766] ? do_raw_spin_unlock+0x75/0x190 [ 4132.817021] ? _raw_spin_unlock+0x12/0x30 [ 4132.821854] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4132.837404] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4132.839233] ll_atomic_open+0x36a/0xd70 [lustre] [ 4132.853000] lookup_open+0x356/0xb40 [ 4132.859240] do_last+0x313/0xfc0 [ 4132.863679] ? nd_jump_root+0xe5/0x160 [ 4132.868739] ? path_init+0x437/0x520 [ 4132.872902] path_openat+0xf7/0x500 [ 4132.876954] ? do_raw_spin_unlock+0x75/0x190 [ 4132.881507] do_filp_open+0x99/0x140 [ 4132.885278] ? getname_flags+0x6e/0x330 [ 4132.889246] ? __check_object_size+0xff/0x256 [ 4132.898888] ? do_raw_spin_unlock+0x75/0x190 [ 4132.905458] ? _raw_spin_unlock+0x12/0x30 [ 4132.915250] do_sys_openat2+0x2b4/0x410 [ 4132.921357] do_sys_open+0x73/0xa0 [ 4132.925961] __x64_sys_openat+0x24/0x30 [ 4132.937167] do_syscall_64+0xc1/0x440 [ 4132.938477] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4132.948162] RIP: 0033:0x7fdc0afbf332 [ 4132.953916] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4132.996889] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4133.009624] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4133.023978] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4133.040647] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4133.044778] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4133.053404] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4134.371508] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2109s! [ 4134.376667] Showing busy workqueues and worker pools: [ 4134.383205] workqueue events: flags=0x0 [ 4134.385257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4134.389178] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4134.402605] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4134.413649] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4134.439252] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4134.454652] pending: kfree_rcu_monitor [ 4134.464355] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4134.475693] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4134.499303] workqueue events_long: flags=0x0 [ 4134.506249] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4134.516333] pending: lu_objects_flush [obdclass] [ 4134.528116] workqueue events_unbound: flags=0x2 [ 4134.529696] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4134.540347] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4134.559878] workqueue events_power_efficient: flags=0x80 [ 4134.565526] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4134.573778] pending: neigh_periodic_work, check_lifetime [ 4134.575706] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4134.585530] pending: do_cache_clean, neigh_periodic_work [ 4134.595274] workqueue mm_percpu_wq: flags=0x8 [ 4134.599911] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4134.605896] pending: lru_add_drain_per_cpu [ 4134.621334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4134.635639] pending: vmstat_update, lru_add_drain_per_cpu [ 4134.640285] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4134.647201] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4134.668096] workqueue writeback: flags=0x4a [ 4134.669690] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4134.689534] pending: wb_workfn, wb_workfn [ 4134.691994] workqueue xprtiod: flags=0xa [ 4134.696293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4134.701918] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4134.716390] workqueue ipv6_addrconf: flags=0x40008 [ 4134.721359] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4134.732403] pending: addrconf_verify_work [ 4134.744895] workqueue knbd0-recv: flags=0x1a [ 4134.749784] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4134.762216] in-flight: 655:recv_work [ 4134.765275] workqueue ptlrpc_pinger: flags=0x2 [ 4134.769661] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4134.779411] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4134.789565] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2109s workers=4 idle: 36396 38 11188 [ 4134.792263] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4125s workers=2 idle: 658 [ 4140.068854] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4140.081682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4140.157670] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4140.178291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4140.194539] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4140.196741] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4140.228523] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4140.232887] RAX: 0000000000007750 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4140.248317] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4140.257173] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4140.270421] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4140.280918] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4140.288415] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4140.299586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4140.308455] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4140.316995] Call Trace: [ 4140.319164] [ 4140.321711] ? show_regs.cold.9+0x22/0x2f [ 4140.327105] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4140.336770] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4140.347338] ? watchdog+0x40/0x40 [ 4140.349415] ? hrtimer_interrupt+0x128/0x2c0 [ 4140.353300] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4140.356206] ? apic_timer_interrupt+0xf/0x20 [ 4140.359377] [ 4140.360100] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4140.370137] do_raw_spin_lock+0x177/0x190 [ 4140.381095] _raw_spin_lock+0x21/0x30 [ 4140.389128] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4140.394984] ? do_raw_spin_unlock+0x75/0x190 [ 4140.409412] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4140.423407] ? do_wait_intr+0xf0/0xf0 [ 4140.425904] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4140.437144] mdc_close+0x53b/0xd70 [mdc] [ 4140.447998] lmv_close+0x18d/0x490 [lmv] [ 4140.454584] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4140.456561] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4140.470146] ll_md_close+0x274/0x800 [lustre] [ 4140.473907] ll_file_release+0x128/0xc50 [lustre] [ 4140.482083] __fput+0x10d/0x440 [ 4140.489904] ____fput+0x12/0x20 [ 4140.493267] task_work_run+0xc6/0x110 [ 4140.495802] exit_to_usermode_loop+0x1dd/0x1f0 [ 4140.501090] do_syscall_64+0x3d6/0x440 [ 4140.504015] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4140.512927] RIP: 0033:0x7fdc0afbfcd8 [ 4140.514125] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4140.546121] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4140.563634] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4140.567606] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4140.581758] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4140.600285] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4140.613108] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4144.053523] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4144.096670] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4144.198643] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4144.224468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4144.254307] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4144.275723] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4144.292578] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4144.342463] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4144.365314] RAX: 0000000000007ce9 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4144.382501] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4144.403197] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4144.420781] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4144.441597] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4144.460709] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4144.468484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4144.476425] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4144.486789] Call Trace: [ 4144.488697] [ 4144.491601] ? show_regs.cold.9+0x22/0x2f [ 4144.496060] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4144.504558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4144.515464] ? watchdog+0x40/0x40 [ 4144.526978] ? hrtimer_interrupt+0x128/0x2c0 [ 4144.539805] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4144.555739] ? apic_timer_interrupt+0xf/0x20 [ 4144.581620] [ 4144.582445] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4144.598912] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4144.611586] do_raw_spin_lock+0x177/0x190 [ 4144.618692] _raw_spin_lock+0x21/0x30 [ 4144.629055] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4144.641266] process_one_work+0x2c8/0x700 [ 4144.656834] worker_thread+0x296/0x6e0 [ 4144.678483] ? rescuer_thread+0x570/0x570 [ 4144.679819] kthread+0x1d1/0x200 [ 4144.687840] ? set_kthread_struct+0x70/0x70 [ 4144.695826] ret_from_fork+0x1f/0x30 [ 4160.049939] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 4160.053122] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 4160.065365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4160.065449] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4160.065458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4160.065461] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4160.065478] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4160.065483] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4160.065488] RAX: 0000000000006083 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4160.065490] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4160.065493] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4160.065495] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4160.065498] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4160.065502] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4160.065505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4160.065507] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4160.065516] Call Trace: [ 4160.065522] [ 4160.065528] ? show_regs.cold.9+0x22/0x2f [ 4160.065538] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4160.065545] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4160.065550] ? watchdog+0x40/0x40 [ 4160.065554] ? hrtimer_interrupt+0x128/0x2c0 [ 4160.065559] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4160.065568] ? apic_timer_interrupt+0xf/0x20 [ 4160.065573] [ 4160.065576] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4160.065581] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4160.065585] ? dump_stack+0xf6/0x10e [ 4160.065591] do_raw_spin_lock+0x177/0x190 [ 4160.065596] _raw_spin_lock+0x21/0x30 [ 4160.065665] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4160.066264] obd_mod_free+0x46/0x140 [ptlrpc] [ 4160.066465] mdc_commit_open+0x91/0xb0 [mdc] [ 4160.079491] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4160.079564] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4160.079571] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4160.079574] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4160.079587] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4160.079592] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4160.079596] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4160.079609] RDX: 0000000000006f1e RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4160.079612] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4160.079614] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4160.079616] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4160.079619] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4160.079622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4160.079624] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4160.079632] Call Trace: [ 4160.079638] [ 4160.079643] ? show_regs.cold.9+0x22/0x2f [ 4160.079653] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4160.079659] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4160.079663] ? watchdog+0x40/0x40 [ 4160.079668] ? hrtimer_interrupt+0x128/0x2c0 [ 4160.079672] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4160.079680] ? apic_timer_interrupt+0xf/0x20 [ 4160.079685] [ 4160.079687] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4160.079692] do_raw_spin_lock+0x177/0x190 [ 4160.079696] _raw_spin_lock+0x21/0x30 [ 4160.079702] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4160.080149] ? do_raw_spin_unlock+0x75/0x190 [ 4160.080155] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4160.080491] ? do_wait_intr+0xf0/0xf0 [ 4160.080500] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4160.080775] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4160.081183] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4160.081696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.081902] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4160.081945] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4160.082122] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4160.082152] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.082235] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4160.082564] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4160.082591] lmv_intent_open+0x338/0x1150 [lmv] [ 4160.082695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.082782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.085229] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4160.085266] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4160.141620] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4160.141911] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4160.142171] after_reply+0xe2c/0x1780 [ptlrpc] [ 4160.142448] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4160.142657] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4160.142862] ? do_wait_intr+0xf0/0xf0 [ 4160.142874] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4160.144176] kthread+0x1d1/0x200 [ 4160.149589] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.149720] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4160.149811] ? do_raw_spin_unlock+0x75/0x190 [ 4160.149819] ? _raw_spin_unlock+0x12/0x30 [ 4160.149826] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4160.149852] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4160.149872] ll_atomic_open+0x36a/0xd70 [lustre] [ 4160.149957] lookup_open+0x356/0xb40 [ 4160.149967] do_last+0x313/0xfc0 [ 4160.149973] ? nd_jump_root+0xe5/0x160 [ 4160.149978] ? path_init+0x437/0x520 [ 4160.149982] path_openat+0xf7/0x500 [ 4160.149987] ? do_raw_spin_unlock+0x75/0x190 [ 4160.149992] do_filp_open+0x99/0x140 [ 4160.150000] ? getname_flags+0x6e/0x330 [ 4160.150004] ? __check_object_size+0xff/0x256 [ 4160.150023] ? do_raw_spin_unlock+0x75/0x190 [ 4160.150027] ? _raw_spin_unlock+0x12/0x30 [ 4160.150032] do_sys_openat2+0x2b4/0x410 [ 4160.150037] do_sys_open+0x73/0xa0 [ 4160.150042] __x64_sys_openat+0x24/0x30 [ 4160.150045] do_syscall_64+0xc1/0x440 [ 4160.150051] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4160.150059] RIP: 0033:0x7fdc0afbf332 [ 4160.150066] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4160.150071] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4160.150076] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4160.150181] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4160.150188] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4160.150190] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4160.150193] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4160.560311] ? set_kthread_struct+0x70/0x70 [ 4160.563149] ret_from_fork+0x1f/0x30 [ 4165.089371] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2140s! [ 4165.101674] Showing busy workqueues and worker pools: [ 4165.107964] workqueue events: flags=0x0 [ 4165.114721] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4165.123665] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4165.130165] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4165.133958] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4165.139962] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4165.144433] pending: kfree_rcu_monitor [ 4165.148781] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4165.162092] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4165.172857] workqueue events_long: flags=0x0 [ 4165.177333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4165.185001] pending: lu_objects_flush [obdclass] [ 4165.189431] workqueue events_unbound: flags=0x2 [ 4165.192719] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4165.204028] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4165.216364] workqueue events_power_efficient: flags=0x80 [ 4165.221527] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4165.231467] pending: neigh_periodic_work, check_lifetime [ 4165.234374] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4165.238606] pending: do_cache_clean, neigh_periodic_work [ 4165.242992] workqueue mm_percpu_wq: flags=0x8 [ 4165.245770] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4165.251788] pending: lru_add_drain_per_cpu [ 4165.260067] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4165.264881] pending: vmstat_update, lru_add_drain_per_cpu [ 4165.268578] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4165.273046] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4165.277941] workqueue writeback: flags=0x4a [ 4165.280941] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4165.284805] pending: wb_workfn, wb_workfn [ 4165.286232] workqueue xprtiod: flags=0xa [ 4165.295014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4165.302506] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4165.323603] workqueue ipv6_addrconf: flags=0x40008 [ 4165.325162] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4165.335423] pending: addrconf_verify_work [ 4165.339462] workqueue knbd0-recv: flags=0x1a [ 4165.343816] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4165.352615] in-flight: 655:recv_work [ 4165.355719] workqueue ptlrpc_pinger: flags=0x2 [ 4165.361785] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4165.369903] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4165.377618] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2140s workers=4 idle: 36396 38 11188 [ 4165.379897] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4155s workers=2 idle: 658 [ 4168.059981] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4168.070385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4168.103720] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4168.109338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4168.113803] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4168.118646] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4168.134194] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4168.138153] RAX: 000000000000648b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4168.144902] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4168.153195] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4168.158359] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4168.161771] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4168.165513] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4168.176137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4168.179261] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4168.181320] Call Trace: [ 4168.182425] [ 4168.182945] ? show_regs.cold.9+0x22/0x2f [ 4168.184643] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4168.188308] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4168.190218] ? watchdog+0x40/0x40 [ 4168.192854] ? hrtimer_interrupt+0x128/0x2c0 [ 4168.194803] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4168.197488] ? apic_timer_interrupt+0xf/0x20 [ 4168.199482] [ 4168.200360] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4168.202985] do_raw_spin_lock+0x177/0x190 [ 4168.204784] _raw_spin_lock+0x21/0x30 [ 4168.206554] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4168.209145] ? do_raw_spin_unlock+0x75/0x190 [ 4168.211948] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4168.214684] ? do_wait_intr+0xf0/0xf0 [ 4168.216203] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4168.219577] mdc_close+0x53b/0xd70 [mdc] [ 4168.221825] lmv_close+0x18d/0x490 [lmv] [ 4168.224913] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4168.227724] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4168.230219] ll_md_close+0x274/0x800 [lustre] [ 4168.232408] ll_file_release+0x128/0xc50 [lustre] [ 4168.234889] __fput+0x10d/0x440 [ 4168.236440] ____fput+0x12/0x20 [ 4168.237965] task_work_run+0xc6/0x110 [ 4168.240152] exit_to_usermode_loop+0x1dd/0x1f0 [ 4168.242556] do_syscall_64+0x3d6/0x440 [ 4168.244560] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4168.247318] RIP: 0033:0x7fdc0afbfcd8 [ 4168.249192] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4168.258667] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4168.262373] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4168.266338] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4168.270151] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4168.273569] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4168.277028] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4172.053337] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4172.067202] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4172.114443] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4172.122325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4172.128373] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4172.131968] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4172.136160] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4172.154106] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4172.158113] RAX: 0000000000007f8b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4172.162223] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4172.165937] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4172.168611] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4172.173037] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4172.177040] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4172.185431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4172.189258] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4172.192985] Call Trace: [ 4172.196002] [ 4172.196916] ? show_regs.cold.9+0x22/0x2f [ 4172.199275] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4172.204584] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4172.206692] ? watchdog+0x40/0x40 [ 4172.210832] ? hrtimer_interrupt+0x128/0x2c0 [ 4172.215216] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4172.217890] ? apic_timer_interrupt+0xf/0x20 [ 4172.219907] [ 4172.220979] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4172.225856] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4172.233285] do_raw_spin_lock+0x177/0x190 [ 4172.239431] _raw_spin_lock+0x21/0x30 [ 4172.242875] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4172.248645] process_one_work+0x2c8/0x700 [ 4172.252538] worker_thread+0x296/0x6e0 [ 4172.254902] ? rescuer_thread+0x570/0x570 [ 4172.257031] kthread+0x1d1/0x200 [ 4172.258344] ? set_kthread_struct+0x70/0x70 [ 4172.260947] ret_from_fork+0x1f/0x30 [ 4188.052195] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4188.062441] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4188.098780] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4188.102550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4188.105949] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4188.108644] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4188.125977] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4188.130378] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4188.133548] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4188.136963] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4188.142996] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4188.147660] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4188.153025] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4188.157363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4188.159911] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4188.163658] Call Trace: [ 4188.164701] [ 4188.166619] ? show_regs.cold.9+0x22/0x2f [ 4188.169060] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4188.173675] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4188.175633] ? watchdog+0x40/0x40 [ 4188.177671] ? hrtimer_interrupt+0x128/0x2c0 [ 4188.180049] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4188.183239] ? apic_timer_interrupt+0xf/0x20 [ 4188.186207] [ 4188.187381] ? native_safe_halt+0x2e/0x50 [ 4188.189541] kvm_wait+0xcd/0xe0 [ 4188.193926] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4188.196636] do_raw_spin_lock+0x177/0x190 [ 4188.199646] _raw_spin_lock+0x21/0x30 [ 4188.201767] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4188.204471] ? do_raw_spin_unlock+0x75/0x190 [ 4188.206650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4188.210693] ? do_wait_intr+0xf0/0xf0 [ 4188.212194] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4188.220502] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4188.224802] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4188.228502] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.238652] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4188.244738] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4188.246183] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4188.247446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.251706] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4188.258214] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4188.263948] lmv_intent_open+0x338/0x1150 [lmv] [ 4188.273076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.277981] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.283553] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4188.286679] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4188.289439] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.291688] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4188.303884] ? do_raw_spin_unlock+0x75/0x190 [ 4188.306975] ? _raw_spin_unlock+0x12/0x30 [ 4188.308205] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4188.313639] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4188.315453] ll_atomic_open+0x36a/0xd70 [lustre] [ 4188.318863] lookup_open+0x356/0xb40 [ 4188.320431] do_last+0x313/0xfc0 [ 4188.322271] ? nd_jump_root+0xe5/0x160 [ 4188.323749] ? path_init+0x437/0x520 [ 4188.325541] path_openat+0xf7/0x500 [ 4188.327537] ? do_raw_spin_unlock+0x75/0x190 [ 4188.328926] do_filp_open+0x99/0x140 [ 4188.330006] ? getname_flags+0x6e/0x330 [ 4188.331994] ? __check_object_size+0xff/0x256 [ 4188.336165] ? do_raw_spin_unlock+0x75/0x190 [ 4188.337384] ? _raw_spin_unlock+0x12/0x30 [ 4188.340813] do_sys_openat2+0x2b4/0x410 [ 4188.343375] do_sys_open+0x73/0xa0 [ 4188.347103] __x64_sys_openat+0x24/0x30 [ 4188.350650] do_syscall_64+0xc1/0x440 [ 4188.356481] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4188.360947] RIP: 0033:0x7fdc0afbf332 [ 4188.364707] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4188.377459] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4188.381811] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4188.385508] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4188.388781] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4188.394903] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4188.397752] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4192.048116] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 4192.054072] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4192.084831] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4192.089985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4192.093769] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4192.096033] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4192.104486] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4192.107425] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4192.110360] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4192.112720] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4192.115353] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4192.118576] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4192.122438] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4192.125634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4192.128423] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4192.131467] Call Trace: [ 4192.132747] [ 4192.134136] ? show_regs.cold.9+0x22/0x2f [ 4192.135999] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4192.138351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4192.140102] ? watchdog+0x40/0x40 [ 4192.144478] ? hrtimer_interrupt+0x128/0x2c0 [ 4192.146909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4192.149577] ? apic_timer_interrupt+0xf/0x20 [ 4192.151831] [ 4192.153129] ? native_safe_halt+0x2e/0x50 [ 4192.160137] kvm_wait+0xcd/0xe0 [ 4192.163977] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4192.170104] ? dump_stack+0xf6/0x10e [ 4192.173891] do_raw_spin_lock+0x177/0x190 [ 4192.178055] _raw_spin_lock+0x21/0x30 [ 4192.183698] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4192.187836] obd_mod_free+0x46/0x140 [ptlrpc] [ 4192.190576] mdc_commit_open+0x91/0xb0 [mdc] [ 4192.193230] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4192.195841] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4192.198180] after_reply+0xe2c/0x1780 [ptlrpc] [ 4192.204613] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4192.207729] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4192.209852] ? do_wait_intr+0xf0/0xf0 [ 4192.216218] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4192.218553] kthread+0x1d1/0x200 [ 4192.220156] ? set_kthread_struct+0x70/0x70 [ 4192.222150] ret_from_fork+0x1f/0x30 [ 4195.813682] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2170s! [ 4195.821635] Showing busy workqueues and worker pools: [ 4195.823595] workqueue events: flags=0x0 [ 4195.826420] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4195.832139] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4195.836500] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4195.838907] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4195.844287] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4195.847908] pending: kfree_rcu_monitor [ 4195.849934] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4195.852912] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4195.861921] workqueue events_long: flags=0x0 [ 4195.863595] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4195.872665] pending: lu_objects_flush [obdclass] [ 4195.874939] workqueue events_unbound: flags=0x2 [ 4195.877137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4195.880166] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4195.888531] workqueue events_power_efficient: flags=0x80 [ 4195.892812] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4195.907388] pending: neigh_periodic_work, check_lifetime [ 4195.916653] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4195.923634] pending: do_cache_clean, neigh_periodic_work [ 4195.925956] workqueue mm_percpu_wq: flags=0x8 [ 4195.927936] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4195.931525] pending: lru_add_drain_per_cpu [ 4195.932984] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4195.935703] pending: vmstat_update, lru_add_drain_per_cpu [ 4195.945362] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4195.948766] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4195.956775] workqueue writeback: flags=0x4a [ 4195.958913] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4195.961841] pending: wb_workfn, wb_workfn [ 4195.964585] workqueue xprtiod: flags=0xa [ 4195.967355] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4195.973583] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4195.979819] workqueue ipv6_addrconf: flags=0x40008 [ 4195.984093] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4195.987123] pending: addrconf_verify_work [ 4195.990430] workqueue knbd0-recv: flags=0x1a [ 4195.992540] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4195.996992] in-flight: 655:recv_work [ 4195.998712] workqueue ptlrpc_pinger: flags=0x2 [ 4196.001950] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4196.009431] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4196.016125] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2171s workers=4 idle: 36396 38 11188 [ 4196.023189] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4186s workers=2 idle: 658 [ 4196.055350] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4196.063350] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4196.111587] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4196.118688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4196.123897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4196.129861] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4196.142943] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4196.145767] RAX: 0000000000007316 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4196.148367] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4196.153784] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4196.159337] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4196.163546] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4196.169870] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4196.175781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4196.177409] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4196.184747] Call Trace: [ 4196.185830] [ 4196.186753] ? show_regs.cold.9+0x22/0x2f [ 4196.188934] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4196.190872] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4196.194419] ? watchdog+0x40/0x40 [ 4196.196073] ? hrtimer_interrupt+0x128/0x2c0 [ 4196.198278] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4196.200817] ? apic_timer_interrupt+0xf/0x20 [ 4196.205962] [ 4196.207897] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4196.211393] do_raw_spin_lock+0x177/0x190 [ 4196.213999] _raw_spin_lock+0x21/0x30 [ 4196.216504] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4196.220339] ? do_raw_spin_unlock+0x75/0x190 [ 4196.222922] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4196.230670] ? do_wait_intr+0xf0/0xf0 [ 4196.233460] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4196.235964] mdc_close+0x53b/0xd70 [mdc] [ 4196.237927] lmv_close+0x18d/0x490 [lmv] [ 4196.244636] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4196.255496] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4196.270540] ll_md_close+0x274/0x800 [lustre] [ 4196.277255] ll_file_release+0x128/0xc50 [lustre] [ 4196.287157] __fput+0x10d/0x440 [ 4196.291388] ____fput+0x12/0x20 [ 4196.296551] task_work_run+0xc6/0x110 [ 4196.300628] exit_to_usermode_loop+0x1dd/0x1f0 [ 4196.302937] do_syscall_64+0x3d6/0x440 [ 4196.304107] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4196.310615] RIP: 0033:0x7fdc0afbfcd8 [ 4196.319030] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4196.333577] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4196.344187] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4196.350405] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4196.353777] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4196.357447] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4196.364690] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4200.051072] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4200.053828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4200.087898] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4200.092514] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4200.095994] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4200.098688] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4200.102954] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4200.114131] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4200.117889] RAX: 0000000000007e77 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4200.126563] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4200.134041] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4200.147190] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4200.149593] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4200.154435] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4200.159732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4200.164009] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4200.169507] Call Trace: [ 4200.171199] [ 4200.171764] ? show_regs.cold.9+0x22/0x2f [ 4200.173217] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4200.174507] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4200.179450] ? watchdog+0x40/0x40 [ 4200.183563] ? hrtimer_interrupt+0x128/0x2c0 [ 4200.186180] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4200.187202] ? apic_timer_interrupt+0xf/0x20 [ 4200.188491] [ 4200.189809] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4200.191519] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4200.194368] do_raw_spin_lock+0x177/0x190 [ 4200.196191] _raw_spin_lock+0x21/0x30 [ 4200.197580] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4200.206812] process_one_work+0x2c8/0x700 [ 4200.209395] worker_thread+0x296/0x6e0 [ 4200.211882] ? rescuer_thread+0x570/0x570 [ 4200.213700] kthread+0x1d1/0x200 [ 4200.214774] ? set_kthread_struct+0x70/0x70 [ 4200.218430] ret_from_fork+0x1f/0x30 [ 4216.055187] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4216.060392] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4216.113218] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4216.120403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4216.125385] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4216.129426] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4216.146529] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4216.159703] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4216.166817] RDX: 0000000000000f4d RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4216.169607] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4216.176313] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4216.181260] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4216.194865] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4216.197621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4216.212668] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4216.222342] Call Trace: [ 4216.223283] [ 4216.224711] ? show_regs.cold.9+0x22/0x2f [ 4216.227199] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4216.230259] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4216.232354] ? watchdog+0x40/0x40 [ 4216.233442] ? hrtimer_interrupt+0x128/0x2c0 [ 4216.235827] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4216.236971] ? apic_timer_interrupt+0xf/0x20 [ 4216.238403] [ 4216.240430] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4216.243565] do_raw_spin_lock+0x177/0x190 [ 4216.250392] _raw_spin_lock+0x21/0x30 [ 4216.256214] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4216.264433] ? do_raw_spin_unlock+0x75/0x190 [ 4216.274051] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4216.275824] ? do_wait_intr+0xf0/0xf0 [ 4216.281834] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4216.286401] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4216.295176] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4216.302591] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.304605] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4216.307572] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4216.309593] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4216.316270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.324687] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4216.329372] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4216.334462] lmv_intent_open+0x338/0x1150 [lmv] [ 4216.335967] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.338433] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.346631] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4216.348126] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4216.350469] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.353008] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4216.355094] ? do_raw_spin_unlock+0x75/0x190 [ 4216.356888] ? _raw_spin_unlock+0x12/0x30 [ 4216.358906] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4216.361297] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4216.363735] ll_atomic_open+0x36a/0xd70 [lustre] [ 4216.365589] lookup_open+0x356/0xb40 [ 4216.367405] do_last+0x313/0xfc0 [ 4216.368325] ? nd_jump_root+0xe5/0x160 [ 4216.369709] ? path_init+0x437/0x520 [ 4216.371441] path_openat+0xf7/0x500 [ 4216.373303] ? do_raw_spin_unlock+0x75/0x190 [ 4216.375286] do_filp_open+0x99/0x140 [ 4216.376374] ? getname_flags+0x6e/0x330 [ 4216.377581] ? __check_object_size+0xff/0x256 [ 4216.380387] ? do_raw_spin_unlock+0x75/0x190 [ 4216.382470] ? _raw_spin_unlock+0x12/0x30 [ 4216.384781] do_sys_openat2+0x2b4/0x410 [ 4216.388034] do_sys_open+0x73/0xa0 [ 4216.392058] __x64_sys_openat+0x24/0x30 [ 4216.393400] do_syscall_64+0xc1/0x440 [ 4216.395065] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4216.400252] RIP: 0033:0x7fdc0afbf332 [ 4216.401509] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4216.412290] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4216.415488] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4216.417920] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4216.423064] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4216.429202] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4216.432543] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4220.050561] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 4220.065761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4220.113367] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4220.118590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4220.130280] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4220.134055] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4220.147683] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4220.152096] RAX: 0000000000003a40 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4220.156994] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4220.162992] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4220.166260] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4220.172076] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4220.182441] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4220.184842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4220.194426] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4220.197104] Call Trace: [ 4220.198094] [ 4220.199077] ? show_regs.cold.9+0x22/0x2f [ 4220.201012] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4220.203667] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4220.205481] ? watchdog+0x40/0x40 [ 4220.207274] ? hrtimer_interrupt+0x128/0x2c0 [ 4220.209151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4220.214455] ? apic_timer_interrupt+0xf/0x20 [ 4220.217306] [ 4220.218267] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4220.224140] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4220.232060] ? dump_stack+0xf6/0x10e [ 4220.235502] do_raw_spin_lock+0x177/0x190 [ 4220.237974] _raw_spin_lock+0x21/0x30 [ 4220.239966] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4220.242679] obd_mod_free+0x46/0x140 [ptlrpc] [ 4220.245224] mdc_commit_open+0x91/0xb0 [mdc] [ 4220.248648] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4220.251252] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4220.258064] after_reply+0xe2c/0x1780 [ptlrpc] [ 4220.261528] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4220.265182] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4220.267157] ? do_wait_intr+0xf0/0xf0 [ 4220.268993] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4220.271283] kthread+0x1d1/0x200 [ 4220.273231] ? set_kthread_struct+0x70/0x70 [ 4220.274911] ret_from_fork+0x1f/0x30 [ 4224.054173] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4224.059400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4224.086470] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4224.094629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4224.100828] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4224.102618] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4224.110699] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4224.114720] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4224.117667] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4224.120308] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4224.122783] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4224.133960] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4224.139070] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4224.143860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4224.146848] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4224.149643] Call Trace: [ 4224.150824] [ 4224.151624] ? show_regs.cold.9+0x22/0x2f [ 4224.153339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4224.155651] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4224.157958] ? watchdog+0x40/0x40 [ 4224.159109] ? hrtimer_interrupt+0x128/0x2c0 [ 4224.161672] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4224.164182] ? apic_timer_interrupt+0xf/0x20 [ 4224.166316] [ 4224.167963] ? native_safe_halt+0x2e/0x50 [ 4224.173904] kvm_wait+0xcd/0xe0 [ 4224.176671] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4224.182431] do_raw_spin_lock+0x177/0x190 [ 4224.185647] _raw_spin_lock+0x21/0x30 [ 4224.187856] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4224.191814] ? do_raw_spin_unlock+0x75/0x190 [ 4224.194445] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4224.197923] ? do_wait_intr+0xf0/0xf0 [ 4224.199836] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4224.203384] mdc_close+0x53b/0xd70 [mdc] [ 4224.211330] lmv_close+0x18d/0x490 [lmv] [ 4224.212504] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4224.227218] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4224.231099] ll_md_close+0x274/0x800 [lustre] [ 4224.234814] ll_file_release+0x128/0xc50 [lustre] [ 4224.240178] __fput+0x10d/0x440 [ 4224.243634] ____fput+0x12/0x20 [ 4224.245374] task_work_run+0xc6/0x110 [ 4224.247366] exit_to_usermode_loop+0x1dd/0x1f0 [ 4224.249308] do_syscall_64+0x3d6/0x440 [ 4224.251388] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4224.254211] RIP: 0033:0x7fdc0afbfcd8 [ 4224.255968] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4224.265844] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4224.270108] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4224.273608] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4224.277192] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4224.280907] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4224.284394] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4226.527154] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2201s! [ 4226.533981] Showing busy workqueues and worker pools: [ 4226.537013] workqueue events: flags=0x0 [ 4226.539230] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4226.542728] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4226.545646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4226.548819] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4226.554494] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4226.557961] pending: kfree_rcu_monitor [ 4226.560373] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4226.563830] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4226.570514] workqueue events_long: flags=0x0 [ 4226.572817] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4226.576571] pending: lu_objects_flush [obdclass] [ 4226.579404] workqueue events_unbound: flags=0x2 [ 4226.581736] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4226.585211] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4226.591099] workqueue events_power_efficient: flags=0x80 [ 4226.594542] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4226.597321] pending: neigh_periodic_work, check_lifetime [ 4226.599710] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4226.603571] pending: do_cache_clean, neigh_periodic_work [ 4226.606228] workqueue mm_percpu_wq: flags=0x8 [ 4226.608356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4226.612120] pending: lru_add_drain_per_cpu [ 4226.614699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4226.617914] pending: vmstat_update, lru_add_drain_per_cpu [ 4226.620947] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4226.624213] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4226.627287] workqueue writeback: flags=0x4a [ 4226.629303] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4226.632270] pending: wb_workfn, wb_workfn [ 4226.634708] workqueue xprtiod: flags=0xa [ 4226.636750] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4226.640036] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4226.643574] workqueue ipv6_addrconf: flags=0x40008 [ 4226.646926] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4226.651544] pending: addrconf_verify_work [ 4226.653646] workqueue knbd0-recv: flags=0x1a [ 4226.656064] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4226.659224] in-flight: 655:recv_work [ 4226.661534] workqueue ptlrpc_pinger: flags=0x2 [ 4226.663560] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4226.666399] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4226.669447] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2201s workers=4 idle: 36396 38 11188 [ 4226.674796] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4217s workers=2 idle: 658 [ 4228.051208] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4228.055569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4228.082745] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4228.088338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4228.092600] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4228.096395] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4228.100565] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 4228.111259] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4228.115648] RAX: 0000000000006a87 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4228.119876] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4228.123491] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4228.126824] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4228.130489] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4228.134150] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4228.138893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4228.141955] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4228.147551] Call Trace: [ 4228.149356] [ 4228.150518] ? show_regs.cold.9+0x22/0x2f [ 4228.152591] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4228.154967] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4228.157235] ? watchdog+0x40/0x40 [ 4228.159149] ? hrtimer_interrupt+0x128/0x2c0 [ 4228.161468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4228.163598] ? apic_timer_interrupt+0xf/0x20 [ 4228.165832] [ 4228.166811] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4228.169440] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4228.171262] do_raw_spin_lock+0x177/0x190 [ 4228.172896] _raw_spin_lock+0x21/0x30 [ 4228.174435] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4228.176668] process_one_work+0x2c8/0x700 [ 4228.179280] worker_thread+0x296/0x6e0 [ 4228.181236] ? rescuer_thread+0x570/0x570 [ 4228.184219] kthread+0x1d1/0x200 [ 4228.186355] ? set_kthread_struct+0x70/0x70 [ 4228.188106] ret_from_fork+0x1f/0x30 [ 4239.391212] rcu: INFO: rcu_sched self-detected stall on CPU [ 4239.394846] rcu: 0-...!: (1298904 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 4239.401474] (t=2220048 jiffies g=270761 q=3279) [ 4239.406973] rcu: rcu_sched kthread starved for 2214980 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 4239.414060] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4239.418941] rcu: RCU grace-period kthread stack dump: [ 4239.421129] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4239.425098] Call Trace: [ 4239.426451] __schedule+0x351/0xcb0 [ 4239.427741] schedule+0xc0/0x180 [ 4239.428955] schedule_timeout+0xb4/0x190 [ 4239.430852] ? do_raw_spin_unlock+0x75/0x190 [ 4239.432670] ? __next_timer_interrupt+0x160/0x160 [ 4239.434527] ? prepare_to_swait_event+0xf3/0x140 [ 4239.436737] rcu_gp_kthread+0x8dd/0x15b0 [ 4239.438192] ? force_qs_rnp+0x2c0/0x2c0 [ 4239.439514] kthread+0x1d1/0x200 [ 4239.441046] ? set_kthread_struct+0x70/0x70 [ 4239.442824] ret_from_fork+0x1f/0x30 [ 4239.444160] rcu: Stack dump where RCU GP kthread last ran: [ 4239.446603] NMI backtrace for cpu 0 [ 4239.447945] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4239.452591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4239.456022] Call Trace: [ 4239.456825] [ 4239.457499] ? dump_stack+0xbb/0x10e [ 4239.459019] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4239.461139] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4239.462886] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4239.464961] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4239.467154] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4239.468913] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 4239.470671] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 4239.472780] ? update_process_times+0x6d/0xc0 [ 4239.474175] ? tick_sched_handle.isra.18+0x27/0x90 [ 4239.475605] ? tick_sched_timer+0x53/0xb0 [ 4239.477145] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4239.478703] ? tick_sched_handle.isra.18+0x90/0x90 [ 4239.480428] ? hrtimer_interrupt+0x128/0x2c0 [ 4239.482050] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4239.483584] ? apic_timer_interrupt+0xf/0x20 [ 4239.484993] [ 4239.486196] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4239.487877] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4239.489909] ? dump_stack+0xf6/0x10e [ 4239.491439] ? do_raw_spin_lock+0x177/0x190 [ 4239.492834] ? _raw_spin_lock+0x21/0x30 [ 4239.494004] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4239.496329] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4239.498489] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4239.500429] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4239.503063] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4239.505586] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4239.507388] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4239.509662] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4239.511550] ? do_wait_intr+0xf0/0xf0 [ 4239.512946] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4239.515463] ? kthread+0x1d1/0x200 [ 4239.516793] ? set_kthread_struct+0x70/0x70 [ 4239.518533] ? ret_from_fork+0x1f/0x30 [ 4239.520004] NMI backtrace for cpu 0 [ 4239.521161] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4239.525528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4239.528818] Call Trace: [ 4239.529699] [ 4239.530950] ? dump_stack+0xbb/0x10e [ 4239.532625] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4239.534554] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4239.536602] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4239.538978] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4239.540975] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4239.542821] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4239.544907] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4239.549003] ? update_process_times+0x6d/0xc0 [ 4239.551559] ? tick_sched_handle.isra.18+0x27/0x90 [ 4239.553506] ? tick_sched_timer+0x53/0xb0 [ 4239.555109] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4239.556731] ? tick_sched_handle.isra.18+0x90/0x90 [ 4239.558818] ? hrtimer_interrupt+0x128/0x2c0 [ 4239.560415] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4239.562333] ? apic_timer_interrupt+0xf/0x20 [ 4239.563936] [ 4239.564778] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4239.567018] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4239.569403] ? dump_stack+0xf6/0x10e [ 4239.570836] ? do_raw_spin_lock+0x177/0x190 [ 4239.572492] ? _raw_spin_lock+0x21/0x30 [ 4239.573887] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4239.576114] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4239.578101] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4239.580127] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4239.583518] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4239.590757] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4239.593103] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4239.595486] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4239.597118] ? do_wait_intr+0xf0/0xf0 [ 4239.599066] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4239.601180] ? kthread+0x1d1/0x200 [ 4239.602301] ? set_kthread_struct+0x70/0x70 [ 4239.603577] ? ret_from_fork+0x1f/0x30 [ 4244.052239] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4244.062360] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4244.088796] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4244.093037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4244.096068] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4244.098417] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4244.105114] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4244.111208] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4244.114805] RDX: 0000000000004350 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4244.118618] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4244.122040] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4244.125555] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4244.129455] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4244.132819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4244.134740] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4244.137199] Call Trace: [ 4244.138482] [ 4244.139092] ? show_regs.cold.9+0x22/0x2f [ 4244.141027] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4244.142726] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4244.144887] ? watchdog+0x40/0x40 [ 4244.146871] ? hrtimer_interrupt+0x128/0x2c0 [ 4244.150757] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4244.153925] ? apic_timer_interrupt+0xf/0x20 [ 4244.156166] [ 4244.157049] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4244.159527] do_raw_spin_lock+0x177/0x190 [ 4244.160873] _raw_spin_lock+0x21/0x30 [ 4244.162611] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4244.164789] ? do_raw_spin_unlock+0x75/0x190 [ 4244.167119] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4244.169861] ? do_wait_intr+0xf0/0xf0 [ 4244.173578] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4244.179702] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4244.186477] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4244.196257] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.200089] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4244.204521] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4244.207313] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4244.211277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.214739] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4244.221030] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4244.224083] lmv_intent_open+0x338/0x1150 [lmv] [ 4244.228385] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.232743] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.235724] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4244.238435] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4244.241115] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.243798] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4244.247336] ? do_raw_spin_unlock+0x75/0x190 [ 4244.250409] ? _raw_spin_unlock+0x12/0x30 [ 4244.252514] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4244.254693] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4244.257017] ll_atomic_open+0x36a/0xd70 [lustre] [ 4244.259318] lookup_open+0x356/0xb40 [ 4244.261091] do_last+0x313/0xfc0 [ 4244.262421] ? nd_jump_root+0xe5/0x160 [ 4244.264080] ? path_init+0x437/0x520 [ 4244.265720] path_openat+0xf7/0x500 [ 4244.268262] ? do_raw_spin_unlock+0x75/0x190 [ 4244.270043] do_filp_open+0x99/0x140 [ 4244.271693] ? getname_flags+0x6e/0x330 [ 4244.273480] ? __check_object_size+0xff/0x256 [ 4244.275495] ? do_raw_spin_unlock+0x75/0x190 [ 4244.277616] ? _raw_spin_unlock+0x12/0x30 [ 4244.279597] do_sys_openat2+0x2b4/0x410 [ 4244.281424] do_sys_open+0x73/0xa0 [ 4244.282723] __x64_sys_openat+0x24/0x30 [ 4244.284486] do_syscall_64+0xc1/0x440 [ 4244.286051] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4244.288297] RIP: 0033:0x7fdc0afbf332 [ 4244.289818] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4244.298974] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4244.302499] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4244.307005] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4244.309509] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4244.316078] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4244.319794] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4252.055529] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4252.063505] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4252.087974] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4252.093620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4252.099203] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4252.108125] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4252.123605] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4252.125936] RAX: 0000000000007244 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4252.129884] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4252.133973] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4252.136869] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4252.139816] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4252.144937] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4252.148897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4252.152066] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4252.155598] Call Trace: [ 4252.156335] [ 4252.156940] ? show_regs.cold.9+0x22/0x2f [ 4252.159127] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4252.161950] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4252.163696] ? watchdog+0x40/0x40 [ 4252.164716] ? hrtimer_interrupt+0x128/0x2c0 [ 4252.166205] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4252.168113] ? apic_timer_interrupt+0xf/0x20 [ 4252.170195] [ 4252.171156] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4252.173797] do_raw_spin_lock+0x177/0x190 [ 4252.179157] _raw_spin_lock+0x21/0x30 [ 4252.181288] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4252.188691] ? do_raw_spin_unlock+0x75/0x190 [ 4252.191699] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4252.194895] ? do_wait_intr+0xf0/0xf0 [ 4252.196721] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4252.199457] mdc_close+0x53b/0xd70 [mdc] [ 4252.200854] lmv_close+0x18d/0x490 [lmv] [ 4252.204936] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4252.211317] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4252.216850] ll_md_close+0x274/0x800 [lustre] [ 4252.221899] ll_file_release+0x128/0xc50 [lustre] [ 4252.226411] __fput+0x10d/0x440 [ 4252.228621] ____fput+0x12/0x20 [ 4252.231916] task_work_run+0xc6/0x110 [ 4252.235855] exit_to_usermode_loop+0x1dd/0x1f0 [ 4252.240533] do_syscall_64+0x3d6/0x440 [ 4252.243208] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4252.245650] RIP: 0033:0x7fdc0afbfcd8 [ 4252.247192] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4252.257216] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4252.260705] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4252.263599] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4252.266604] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4252.269621] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4252.272694] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4256.051011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4256.053006] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4256.083681] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4256.094722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4256.100615] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4256.103725] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4256.107603] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4256.127490] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4256.138512] RAX: 00000000000063d9 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4256.148584] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4256.153236] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4256.162905] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4256.167601] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4256.176547] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4256.181020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4256.187419] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4256.195752] Call Trace: [ 4256.198687] [ 4256.199430] ? show_regs.cold.9+0x22/0x2f [ 4256.200909] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4256.203438] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4256.204726] ? watchdog+0x40/0x40 [ 4256.208069] ? hrtimer_interrupt+0x128/0x2c0 [ 4256.210852] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4256.218632] ? apic_timer_interrupt+0xf/0x20 [ 4256.221671] [ 4256.222765] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4256.227812] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4256.235339] do_raw_spin_lock+0x177/0x190 [ 4256.238589] _raw_spin_lock+0x21/0x30 [ 4256.241674] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4256.248415] process_one_work+0x2c8/0x700 [ 4256.251797] worker_thread+0x296/0x6e0 [ 4256.253840] ? rescuer_thread+0x570/0x570 [ 4256.256048] kthread+0x1d1/0x200 [ 4256.259494] ? set_kthread_struct+0x70/0x70 [ 4256.262897] ret_from_fork+0x1f/0x30 [ 4256.736541] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2231s! [ 4256.749585] Showing busy workqueues and worker pools: [ 4256.753328] workqueue events: flags=0x0 [ 4256.758134] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4256.771348] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4256.781142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4256.792148] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4256.801220] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4256.808483] pending: kfree_rcu_monitor [ 4256.812273] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4256.818555] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4256.828766] workqueue events_long: flags=0x0 [ 4256.832047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4256.834770] pending: lu_objects_flush [obdclass] [ 4256.837371] workqueue events_unbound: flags=0x2 [ 4256.839492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4256.842448] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4256.852132] workqueue events_power_efficient: flags=0x80 [ 4256.856339] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4256.863214] pending: neigh_periodic_work, check_lifetime [ 4256.869339] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4256.875044] pending: do_cache_clean, neigh_periodic_work [ 4256.877593] workqueue mm_percpu_wq: flags=0x8 [ 4256.879454] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4256.883073] pending: lru_add_drain_per_cpu [ 4256.884889] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4256.887774] pending: vmstat_update, lru_add_drain_per_cpu [ 4256.890737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4256.898891] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4256.903715] workqueue writeback: flags=0x4a [ 4256.905194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4256.910056] pending: wb_workfn, wb_workfn [ 4256.913210] workqueue xprtiod: flags=0xa [ 4256.917107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4256.919720] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4256.923509] workqueue ipv6_addrconf: flags=0x40008 [ 4256.926442] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4256.931764] pending: addrconf_verify_work [ 4256.941082] workqueue knbd0-recv: flags=0x1a [ 4256.942707] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4256.954747] in-flight: 655:recv_work [ 4256.956896] workqueue ptlrpc_pinger: flags=0x2 [ 4256.959698] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4256.967047] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4256.968775] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2232s workers=4 idle: 36396 38 11188 [ 4256.974981] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4247s workers=2 idle: 658 [ 4272.052036] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4272.055568] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4272.077401] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4272.081828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4272.087326] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4272.090090] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4272.100584] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4272.104229] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4272.110151] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4272.114848] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4272.122451] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4272.126844] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4272.130577] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4272.135001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4272.137831] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4272.141736] Call Trace: [ 4272.142980] [ 4272.143816] ? show_regs.cold.9+0x22/0x2f [ 4272.145926] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4272.148131] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4272.150271] ? watchdog+0x40/0x40 [ 4272.152268] ? hrtimer_interrupt+0x128/0x2c0 [ 4272.155256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4272.157478] ? apic_timer_interrupt+0xf/0x20 [ 4272.160090] [ 4272.161033] ? native_safe_halt+0x2e/0x50 [ 4272.162816] kvm_wait+0xcd/0xe0 [ 4272.164314] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4272.165820] do_raw_spin_lock+0x177/0x190 [ 4272.166991] _raw_spin_lock+0x21/0x30 [ 4272.168777] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4272.174861] ? do_raw_spin_unlock+0x75/0x190 [ 4272.177790] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4272.182256] ? do_wait_intr+0xf0/0xf0 [ 4272.184038] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4272.186806] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4272.189498] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4272.192827] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.196084] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4272.200390] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4272.202287] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4272.203947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.206691] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4272.211792] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4272.218843] lmv_intent_open+0x338/0x1150 [lmv] [ 4272.223438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.232784] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.244968] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4272.247842] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4272.251028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.256933] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4272.258827] ? do_raw_spin_unlock+0x75/0x190 [ 4272.260290] ? _raw_spin_unlock+0x12/0x30 [ 4272.263102] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4272.265569] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4272.269170] ll_atomic_open+0x36a/0xd70 [lustre] [ 4272.274490] lookup_open+0x356/0xb40 [ 4272.279109] do_last+0x313/0xfc0 [ 4272.280260] ? nd_jump_root+0xe5/0x160 [ 4272.281520] ? path_init+0x437/0x520 [ 4272.283055] path_openat+0xf7/0x500 [ 4272.285523] ? do_raw_spin_unlock+0x75/0x190 [ 4272.287782] do_filp_open+0x99/0x140 [ 4272.291871] ? getname_flags+0x6e/0x330 [ 4272.293474] ? __check_object_size+0xff/0x256 [ 4272.295702] ? do_raw_spin_unlock+0x75/0x190 [ 4272.297752] ? _raw_spin_unlock+0x12/0x30 [ 4272.299619] do_sys_openat2+0x2b4/0x410 [ 4272.300787] do_sys_open+0x73/0xa0 [ 4272.302469] __x64_sys_openat+0x24/0x30 [ 4272.303988] do_syscall_64+0xc1/0x440 [ 4272.305951] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4272.308483] RIP: 0033:0x7fdc0afbf332 [ 4272.311210] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4272.319236] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4272.323143] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4272.326398] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4272.329441] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4272.331232] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4272.333696] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4280.056221] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4280.060600] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4280.090457] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4280.099282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4280.105345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4280.108534] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4280.120060] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4280.123688] RAX: 0000000000007109 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4280.126487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4280.128344] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4280.130209] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4280.132445] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4280.134601] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4280.136962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4280.139305] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4280.144596] Call Trace: [ 4280.145678] [ 4280.146326] ? show_regs.cold.9+0x22/0x2f [ 4280.148197] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4280.152224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4280.153960] ? watchdog+0x40/0x40 [ 4280.155826] ? hrtimer_interrupt+0x128/0x2c0 [ 4280.158655] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4280.160951] ? apic_timer_interrupt+0xf/0x20 [ 4280.162769] [ 4280.163702] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4280.166607] do_raw_spin_lock+0x177/0x190 [ 4280.168484] _raw_spin_lock+0x21/0x30 [ 4280.170620] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4280.173326] ? do_raw_spin_unlock+0x75/0x190 [ 4280.175492] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4280.177565] ? do_wait_intr+0xf0/0xf0 [ 4280.178824] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4280.180995] mdc_close+0x53b/0xd70 [mdc] [ 4280.182232] lmv_close+0x18d/0x490 [lmv] [ 4280.183345] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4280.185878] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4280.187827] ll_md_close+0x274/0x800 [lustre] [ 4280.189871] ll_file_release+0x128/0xc50 [lustre] [ 4280.191981] __fput+0x10d/0x440 [ 4280.193813] ____fput+0x12/0x20 [ 4280.195262] task_work_run+0xc6/0x110 [ 4280.196956] exit_to_usermode_loop+0x1dd/0x1f0 [ 4280.200388] do_syscall_64+0x3d6/0x440 [ 4280.202427] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4280.204993] RIP: 0033:0x7fdc0afbfcd8 [ 4280.208007] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4280.227084] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4280.231918] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4280.235276] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4280.238354] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4280.241654] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4280.245055] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4284.048313] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4284.051023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4284.053405] Modules linked in: lustre(O) [ 4284.059079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4284.059160] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4284.059168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4284.059175] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4284.061055] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4284.063095] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4284.063167] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4284.063174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4284.063176] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4284.063188] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4284.063192] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4284.063196] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4284.063199] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4284.063201] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4284.063203] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4284.063205] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4284.063209] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4284.063212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4284.063214] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4284.063426] Call Trace: [ 4284.100079] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4284.111926] [ 4284.115501] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4284.115511] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4284.115513] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4284.115515] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4284.115516] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4284.115518] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4284.115520] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4284.115522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4284.115523] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4284.115529] Call Trace: [ 4284.115534] [ 4284.115539] ? show_regs.cold.9+0x22/0x2f [ 4284.115652] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4284.115657] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4284.115660] ? watchdog+0x40/0x40 [ 4284.115662] ? hrtimer_interrupt+0x128/0x2c0 [ 4284.115665] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4284.115672] ? apic_timer_interrupt+0xf/0x20 [ 4284.115675] [ 4284.115677] ? native_safe_halt+0x2e/0x50 [ 4284.115680] kvm_wait+0xcd/0xe0 [ 4284.115686] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4284.115693] do_raw_spin_lock+0x177/0x190 [ 4284.115697] _raw_spin_lock+0x21/0x30 [ 4284.115699] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4284.116100] process_one_work+0x2c8/0x700 [ 4284.116112] worker_thread+0x296/0x6e0 [ 4284.116116] ? rescuer_thread+0x570/0x570 [ 4284.116120] kthread+0x1d1/0x200 [ 4284.116123] ? set_kthread_struct+0x70/0x70 [ 4284.116127] ret_from_fork+0x1f/0x30 [ 4284.341255] ? show_regs.cold.9+0x22/0x2f [ 4284.349190] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4284.351073] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4284.353699] ? watchdog+0x40/0x40 [ 4284.355090] ? hrtimer_interrupt+0x128/0x2c0 [ 4284.359336] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4284.361000] ? apic_timer_interrupt+0xf/0x20 [ 4284.363450] [ 4284.364232] ? native_safe_halt+0x2e/0x50 [ 4284.369105] kvm_wait+0xcd/0xe0 [ 4284.371304] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4284.378518] ? dump_stack+0xf6/0x10e [ 4284.380115] do_raw_spin_lock+0x177/0x190 [ 4284.382911] _raw_spin_lock+0x21/0x30 [ 4284.384972] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4284.393107] obd_mod_free+0x46/0x140 [ptlrpc] [ 4284.396107] mdc_commit_open+0x91/0xb0 [mdc] [ 4284.400270] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4284.404171] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4284.407063] after_reply+0xe2c/0x1780 [ptlrpc] [ 4284.409143] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4284.411658] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4284.414643] ? do_wait_intr+0xf0/0xf0 [ 4284.416418] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4284.422382] kthread+0x1d1/0x200 [ 4284.424387] ? set_kthread_struct+0x70/0x70 [ 4284.432098] ret_from_fork+0x1f/0x30 [ 4287.455035] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2262s! [ 4287.462114] Showing busy workqueues and worker pools: [ 4287.464281] workqueue events: flags=0x0 [ 4287.466607] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4287.470228] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4287.473306] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4287.476904] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4287.482111] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4287.485271] pending: kfree_rcu_monitor [ 4287.486853] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4287.489929] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4287.495846] workqueue events_long: flags=0x0 [ 4287.497981] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4287.501428] pending: lu_objects_flush [obdclass] [ 4287.504010] workqueue events_unbound: flags=0x2 [ 4287.506018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4287.508791] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4287.513235] workqueue events_power_efficient: flags=0x80 [ 4287.516078] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4287.519861] pending: neigh_periodic_work, check_lifetime [ 4287.522255] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4287.525196] pending: do_cache_clean, neigh_periodic_work [ 4287.528227] workqueue mm_percpu_wq: flags=0x8 [ 4287.530286] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4287.533551] pending: lru_add_drain_per_cpu [ 4287.535705] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4287.539545] pending: vmstat_update, lru_add_drain_per_cpu [ 4287.542734] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4287.546059] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4287.549357] workqueue writeback: flags=0x4a [ 4287.551352] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4287.554457] pending: wb_workfn, wb_workfn [ 4287.556627] workqueue xprtiod: flags=0xa [ 4287.558304] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4287.560892] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4287.564856] workqueue ipv6_addrconf: flags=0x40008 [ 4287.566892] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4287.569211] pending: addrconf_verify_work [ 4287.571096] workqueue knbd0-recv: flags=0x1a [ 4287.573208] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4287.575901] in-flight: 655:recv_work [ 4287.578173] workqueue ptlrpc_pinger: flags=0x2 [ 4287.580254] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4287.582997] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4287.586045] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2262s workers=4 idle: 36396 38 11188 [ 4287.590077] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4278s workers=2 idle: 658 [ 4300.052049] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4300.056695] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4300.085892] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4300.091703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4300.095128] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4300.097191] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4300.105605] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4300.109226] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4300.112003] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4300.116601] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4300.121455] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4300.124077] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4300.126419] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4300.129013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4300.130910] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4300.133241] Call Trace: [ 4300.134099] [ 4300.134923] ? show_regs.cold.9+0x22/0x2f [ 4300.136109] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4300.137646] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4300.139240] ? watchdog+0x40/0x40 [ 4300.140338] ? hrtimer_interrupt+0x128/0x2c0 [ 4300.141837] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4300.143146] ? apic_timer_interrupt+0xf/0x20 [ 4300.144677] [ 4300.145474] ? native_safe_halt+0x2e/0x50 [ 4300.146704] kvm_wait+0xcd/0xe0 [ 4300.147490] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4300.149012] do_raw_spin_lock+0x177/0x190 [ 4300.150201] _raw_spin_lock+0x21/0x30 [ 4300.151218] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4300.153135] ? do_raw_spin_unlock+0x75/0x190 [ 4300.154649] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4300.156800] ? do_wait_intr+0xf0/0xf0 [ 4300.157965] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4300.160024] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4300.162675] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4300.166022] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.168963] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4300.172097] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4300.174574] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4300.176067] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.177934] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4300.180784] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4300.183106] lmv_intent_open+0x338/0x1150 [lmv] [ 4300.185108] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.191765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.194693] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4300.197077] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4300.199612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.203421] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4300.206859] ? do_raw_spin_unlock+0x75/0x190 [ 4300.209634] ? _raw_spin_unlock+0x12/0x30 [ 4300.211193] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4300.214189] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4300.216419] ll_atomic_open+0x36a/0xd70 [lustre] [ 4300.218596] lookup_open+0x356/0xb40 [ 4300.220441] do_last+0x313/0xfc0 [ 4300.221942] ? nd_jump_root+0xe5/0x160 [ 4300.223814] ? path_init+0x437/0x520 [ 4300.225489] path_openat+0xf7/0x500 [ 4300.228627] ? do_raw_spin_unlock+0x75/0x190 [ 4300.231039] do_filp_open+0x99/0x140 [ 4300.233128] ? getname_flags+0x6e/0x330 [ 4300.235628] ? __check_object_size+0xff/0x256 [ 4300.237672] ? do_raw_spin_unlock+0x75/0x190 [ 4300.239661] ? _raw_spin_unlock+0x12/0x30 [ 4300.241576] do_sys_openat2+0x2b4/0x410 [ 4300.243509] do_sys_open+0x73/0xa0 [ 4300.245229] __x64_sys_openat+0x24/0x30 [ 4300.247387] do_syscall_64+0xc1/0x440 [ 4300.249641] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4300.251850] RIP: 0033:0x7fdc0afbf332 [ 4300.253857] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4300.266103] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4300.269024] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4300.272863] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4300.276500] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4300.279963] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4300.283677] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4308.054226] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4308.061232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4308.094913] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4308.099430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4308.105951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4308.108631] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4308.134184] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4308.138426] RAX: 0000000000007e94 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4308.157789] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4308.173401] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4308.184609] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4308.197182] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4308.208458] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4308.211108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4308.221113] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4308.227937] Call Trace: [ 4308.229837] [ 4308.230578] ? show_regs.cold.9+0x22/0x2f [ 4308.234921] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4308.241195] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4308.244404] ? watchdog+0x40/0x40 [ 4308.247329] ? hrtimer_interrupt+0x128/0x2c0 [ 4308.250207] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4308.251608] ? apic_timer_interrupt+0xf/0x20 [ 4308.253823] [ 4308.254804] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4308.256887] do_raw_spin_lock+0x177/0x190 [ 4308.259049] _raw_spin_lock+0x21/0x30 [ 4308.260367] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4308.262773] ? do_raw_spin_unlock+0x75/0x190 [ 4308.264299] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4308.266891] ? do_wait_intr+0xf0/0xf0 [ 4308.271685] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4308.275158] mdc_close+0x53b/0xd70 [mdc] [ 4308.276632] lmv_close+0x18d/0x490 [lmv] [ 4308.278455] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4308.281774] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4308.283469] ll_md_close+0x274/0x800 [lustre] [ 4308.284736] ll_file_release+0x128/0xc50 [lustre] [ 4308.288450] __fput+0x10d/0x440 [ 4308.289823] ____fput+0x12/0x20 [ 4308.291392] task_work_run+0xc6/0x110 [ 4308.299738] exit_to_usermode_loop+0x1dd/0x1f0 [ 4308.302322] do_syscall_64+0x3d6/0x440 [ 4308.303672] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4308.305620] RIP: 0033:0x7fdc0afbfcd8 [ 4308.306682] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4308.313616] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4308.317405] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4308.319272] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4308.322869] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4308.331786] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4308.336675] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4312.048503] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4312.052810] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4312.053051] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4312.072470] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4312.072485] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4312.072488] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.072502] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4312.072506] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4312.072512] RAX: 0000000000001688 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4312.072515] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4312.072517] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4312.072519] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4312.072522] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4312.072525] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4312.072528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4312.072531] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4312.072540] Call Trace: [ 4312.072545] [ 4312.072551] ? show_regs.cold.9+0x22/0x2f [ 4312.072560] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4312.072566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4312.072570] ? watchdog+0x40/0x40 [ 4312.072573] ? hrtimer_interrupt+0x128/0x2c0 [ 4312.072577] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4312.072585] ? apic_timer_interrupt+0xf/0x20 [ 4312.072590] [ 4312.072593] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.072597] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4312.072600] ? dump_stack+0xf6/0x10e [ 4312.072605] do_raw_spin_lock+0x177/0x190 [ 4312.072610] _raw_spin_lock+0x21/0x30 [ 4312.072618] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4312.072890] obd_mod_free+0x46/0x140 [ptlrpc] [ 4312.076490] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4312.076574] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4312.076581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4312.076587] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4312.076857] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.076869] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4312.076873] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4312.076877] RAX: 0000000000007504 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4312.076880] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4312.076882] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4312.076883] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4312.076885] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4312.076888] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4312.076890] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4312.076892] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4312.076902] Call Trace: [ 4312.076908] [ 4312.076912] ? show_regs.cold.9+0x22/0x2f [ 4312.076922] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4312.076928] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4312.076932] ? watchdog+0x40/0x40 [ 4312.076937] ? hrtimer_interrupt+0x128/0x2c0 [ 4312.076941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4312.076949] ? apic_timer_interrupt+0xf/0x20 [ 4312.076954] [ 4312.076957] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.076961] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4312.076966] do_raw_spin_lock+0x177/0x190 [ 4312.076970] _raw_spin_lock+0x21/0x30 [ 4312.076975] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4312.077168] process_one_work+0x2c8/0x700 [ 4312.077176] worker_thread+0x296/0x6e0 [ 4312.077180] ? rescuer_thread+0x570/0x570 [ 4312.077184] kthread+0x1d1/0x200 [ 4312.077188] ? set_kthread_struct+0x70/0x70 [ 4312.077192] ret_from_fork+0x1f/0x30 [ 4312.603280] mdc_commit_open+0x91/0xb0 [mdc] [ 4312.605358] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4312.608655] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4312.612158] after_reply+0xe2c/0x1780 [ptlrpc] [ 4312.614513] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4312.618219] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4312.620629] ? do_wait_intr+0xf0/0xf0 [ 4312.622709] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4312.626908] kthread+0x1d1/0x200 [ 4312.628922] ? set_kthread_struct+0x70/0x70 [ 4312.631617] ret_from_fork+0x1f/0x30 [ 4317.668261] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2292s! [ 4317.676564] Showing busy workqueues and worker pools: [ 4317.684112] workqueue events: flags=0x0 [ 4317.687316] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4317.692600] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4317.702232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4317.710523] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4317.722768] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4317.726227] pending: kfree_rcu_monitor [ 4317.730654] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4317.734435] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4317.744352] workqueue events_long: flags=0x0 [ 4317.748692] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4317.755562] pending: lu_objects_flush [obdclass] [ 4317.764678] workqueue events_unbound: flags=0x2 [ 4317.766403] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4317.778800] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4317.786331] workqueue events_power_efficient: flags=0x80 [ 4317.801194] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4317.808630] pending: neigh_periodic_work, check_lifetime [ 4317.816399] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4317.821803] pending: do_cache_clean, neigh_periodic_work [ 4317.823759] workqueue mm_percpu_wq: flags=0x8 [ 4317.826016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4317.829383] pending: lru_add_drain_per_cpu [ 4317.832792] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4317.836429] pending: vmstat_update, lru_add_drain_per_cpu [ 4317.842745] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4317.845642] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4317.848387] workqueue writeback: flags=0x4a [ 4317.850443] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4317.855092] pending: wb_workfn, wb_workfn [ 4317.858714] workqueue xprtiod: flags=0xa [ 4317.860622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4317.863695] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4317.867300] workqueue ipv6_addrconf: flags=0x40008 [ 4317.869214] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4317.871918] pending: addrconf_verify_work [ 4317.873408] workqueue knbd0-recv: flags=0x1a [ 4317.879431] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4317.881652] in-flight: 655:recv_work [ 4317.884214] workqueue ptlrpc_pinger: flags=0x2 [ 4317.886079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4317.888812] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4317.891582] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2292s workers=4 idle: 36396 38 11188 [ 4317.900026] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4308s workers=2 idle: 658 [ 4328.052008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4328.055592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4328.079597] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4328.089100] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4328.100337] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4328.102988] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4328.113785] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4328.117762] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4328.121956] RDX: 0000000000004fb7 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4328.127961] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4328.132297] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4328.137110] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4328.142527] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4328.146478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4328.148495] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4328.154997] Call Trace: [ 4328.156375] [ 4328.159874] ? show_regs.cold.9+0x22/0x2f [ 4328.162134] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4328.165334] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4328.175205] ? watchdog+0x40/0x40 [ 4328.178798] ? hrtimer_interrupt+0x128/0x2c0 [ 4328.193649] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4328.199480] ? apic_timer_interrupt+0xf/0x20 [ 4328.200697] [ 4328.203470] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4328.210414] do_raw_spin_lock+0x177/0x190 [ 4328.214541] _raw_spin_lock+0x21/0x30 [ 4328.216729] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4328.222301] ? do_raw_spin_unlock+0x75/0x190 [ 4328.225302] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4328.228140] ? do_wait_intr+0xf0/0xf0 [ 4328.234281] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4328.239115] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4328.241542] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4328.254250] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4328.265188] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4328.271298] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4328.278785] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4328.281341] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4328.286356] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4328.291850] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4328.294192] lmv_intent_open+0x338/0x1150 [lmv] [ 4328.297291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4328.309304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4328.315254] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4328.318328] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4328.321664] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4328.327376] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4328.331515] ? do_raw_spin_unlock+0x75/0x190 [ 4328.333768] ? _raw_spin_unlock+0x12/0x30 [ 4328.337410] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4328.345464] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4328.350308] ll_atomic_open+0x36a/0xd70 [lustre] [ 4328.358034] lookup_open+0x356/0xb40 [ 4328.362528] do_last+0x313/0xfc0 [ 4328.365387] ? nd_jump_root+0xe5/0x160 [ 4328.367216] ? path_init+0x437/0x520 [ 4328.373269] path_openat+0xf7/0x500 [ 4328.376567] ? do_raw_spin_unlock+0x75/0x190 [ 4328.381591] do_filp_open+0x99/0x140 [ 4328.385170] ? getname_flags+0x6e/0x330 [ 4328.388275] ? __check_object_size+0xff/0x256 [ 4328.393235] ? do_raw_spin_unlock+0x75/0x190 [ 4328.397358] ? _raw_spin_unlock+0x12/0x30 [ 4328.401640] do_sys_openat2+0x2b4/0x410 [ 4328.405637] do_sys_open+0x73/0xa0 [ 4328.411286] __x64_sys_openat+0x24/0x30 [ 4328.415848] do_syscall_64+0xc1/0x440 [ 4328.421214] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4328.429534] RIP: 0033:0x7fdc0afbf332 [ 4328.434123] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4328.453243] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4328.458756] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4328.463820] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4328.478611] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4328.504138] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4328.518382] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4336.056258] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4336.068452] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4336.131407] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4336.143636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4336.153997] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4336.161664] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4336.179911] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4336.184227] RAX: 0000000000007788 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4336.187434] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4336.195092] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4336.207900] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4336.215743] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4336.222737] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4336.234484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4336.238582] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4336.252287] Call Trace: [ 4336.264468] [ 4336.265185] ? show_regs.cold.9+0x22/0x2f [ 4336.266395] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4336.270963] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4336.272390] ? watchdog+0x40/0x40 [ 4336.283013] ? hrtimer_interrupt+0x128/0x2c0 [ 4336.289094] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4336.292744] ? apic_timer_interrupt+0xf/0x20 [ 4336.305738] [ 4336.309864] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4336.317024] do_raw_spin_lock+0x177/0x190 [ 4336.328704] _raw_spin_lock+0x21/0x30 [ 4336.334766] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4336.336728] ? do_raw_spin_unlock+0x75/0x190 [ 4336.342082] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4336.349215] ? do_wait_intr+0xf0/0xf0 [ 4336.356470] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4336.363187] mdc_close+0x53b/0xd70 [mdc] [ 4336.369214] lmv_close+0x18d/0x490 [lmv] [ 4336.374706] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4336.376800] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4336.395244] ll_md_close+0x274/0x800 [lustre] [ 4336.397276] ll_file_release+0x128/0xc50 [lustre] [ 4336.404897] __fput+0x10d/0x440 [ 4336.409747] ____fput+0x12/0x20 [ 4336.410949] task_work_run+0xc6/0x110 [ 4336.412003] exit_to_usermode_loop+0x1dd/0x1f0 [ 4336.424339] do_syscall_64+0x3d6/0x440 [ 4336.426404] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4336.432240] RIP: 0033:0x7fdc0afbfcd8 [ 4336.435085] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4336.460982] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4336.468066] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4336.470712] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4336.482146] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4336.489543] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4336.497260] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4340.051324] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4340.056288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4340.077046] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4340.081810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4340.085944] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4340.089093] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4340.091814] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4340.099670] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4340.102849] RAX: 0000000000006331 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4340.106106] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4340.109031] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4340.112357] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4340.115167] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4340.118156] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4340.121634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4340.124386] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4340.127332] Call Trace: [ 4340.128753] [ 4340.129657] ? show_regs.cold.9+0x22/0x2f [ 4340.131264] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4340.133374] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4340.134944] ? watchdog+0x40/0x40 [ 4340.136107] ? hrtimer_interrupt+0x128/0x2c0 [ 4340.137344] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4340.138664] ? apic_timer_interrupt+0xf/0x20 [ 4340.140591] [ 4340.141387] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4340.143923] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4340.146273] do_raw_spin_lock+0x177/0x190 [ 4340.147941] _raw_spin_lock+0x21/0x30 [ 4340.149585] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4340.151948] process_one_work+0x2c8/0x700 [ 4340.153639] worker_thread+0x296/0x6e0 [ 4340.155194] ? rescuer_thread+0x570/0x570 [ 4340.156928] kthread+0x1d1/0x200 [ 4340.158065] ? set_kthread_struct+0x70/0x70 [ 4340.160072] ret_from_fork+0x1f/0x30 [ 4344.048033] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4344.052743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4344.072151] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4344.075734] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4344.080571] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4344.081911] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4344.088561] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4344.090879] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4344.093021] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4344.095859] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4344.100111] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4344.103589] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4344.105554] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4344.107482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4344.108915] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4344.110620] Call Trace: [ 4344.111197] [ 4344.111715] ? show_regs.cold.9+0x22/0x2f [ 4344.112947] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4344.114862] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4344.116446] ? watchdog+0x40/0x40 [ 4344.117198] ? hrtimer_interrupt+0x128/0x2c0 [ 4344.118257] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4344.120046] ? apic_timer_interrupt+0xf/0x20 [ 4344.121096] [ 4344.121672] ? native_safe_halt+0x2e/0x50 [ 4344.122636] kvm_wait+0xcd/0xe0 [ 4344.123470] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4344.124697] ? dump_stack+0xf6/0x10e [ 4344.125693] do_raw_spin_lock+0x177/0x190 [ 4344.126801] _raw_spin_lock+0x21/0x30 [ 4344.127735] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4344.130089] obd_mod_free+0x46/0x140 [ptlrpc] [ 4344.131742] mdc_commit_open+0x91/0xb0 [mdc] [ 4344.133170] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4344.136011] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4344.139201] after_reply+0xe2c/0x1780 [ptlrpc] [ 4344.141260] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4344.143348] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4344.145110] ? do_wait_intr+0xf0/0xf0 [ 4344.146223] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4344.148224] kthread+0x1d1/0x200 [ 4344.149427] ? set_kthread_struct+0x70/0x70 [ 4344.150931] ret_from_fork+0x1f/0x30 [ 4348.383072] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2323s! [ 4348.387772] Showing busy workqueues and worker pools: [ 4348.389931] workqueue events: flags=0x0 [ 4348.395070] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4348.399456] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4348.402785] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4348.406745] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4348.416280] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4348.420585] pending: kfree_rcu_monitor [ 4348.422708] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4348.426797] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4348.434352] workqueue events_long: flags=0x0 [ 4348.436540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4348.439898] pending: lu_objects_flush [obdclass] [ 4348.442879] workqueue events_unbound: flags=0x2 [ 4348.445396] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4348.449051] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4348.454146] workqueue events_power_efficient: flags=0x80 [ 4348.456914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4348.460799] pending: neigh_periodic_work, check_lifetime [ 4348.463805] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4348.467612] pending: do_cache_clean, neigh_periodic_work [ 4348.470699] workqueue mm_percpu_wq: flags=0x8 [ 4348.474193] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4348.479112] pending: lru_add_drain_per_cpu [ 4348.481856] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4348.485739] pending: vmstat_update, lru_add_drain_per_cpu [ 4348.489671] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4348.493462] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4348.496839] workqueue writeback: flags=0x4a [ 4348.499184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4348.502368] pending: wb_workfn, wb_workfn [ 4348.504787] workqueue xprtiod: flags=0xa [ 4348.506820] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4348.510089] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4348.514123] workqueue ipv6_addrconf: flags=0x40008 [ 4348.516925] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4348.520461] pending: addrconf_verify_work [ 4348.522907] workqueue knbd0-recv: flags=0x1a [ 4348.524813] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4348.528533] in-flight: 655:recv_work [ 4348.531247] workqueue ptlrpc_pinger: flags=0x2 [ 4348.533379] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4348.536382] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4348.539427] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2323s workers=4 idle: 36396 38 11188 [ 4348.543097] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4339s workers=2 idle: 658 [ 4356.052062] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4356.054214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4356.092483] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4356.099147] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4356.108108] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4356.110149] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4356.129639] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4356.135422] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4356.144867] RDX: 0000000000000aa8 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4356.150147] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4356.158595] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4356.164566] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4356.170943] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4356.173959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4356.175812] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4356.185749] Call Trace: [ 4356.188305] [ 4356.196143] ? show_regs.cold.9+0x22/0x2f [ 4356.201835] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4356.205451] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4356.207481] ? watchdog+0x40/0x40 [ 4356.212811] ? hrtimer_interrupt+0x128/0x2c0 [ 4356.215354] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4356.218237] ? apic_timer_interrupt+0xf/0x20 [ 4356.222361] [ 4356.223255] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4356.225423] do_raw_spin_lock+0x177/0x190 [ 4356.231774] _raw_spin_lock+0x21/0x30 [ 4356.235381] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4356.240129] ? do_raw_spin_unlock+0x75/0x190 [ 4356.243571] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4356.248566] ? do_wait_intr+0xf0/0xf0 [ 4356.252249] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4356.254881] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4356.257190] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4356.260715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.266720] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4356.275430] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4356.277179] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4356.278515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.284702] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4356.288920] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4356.291464] lmv_intent_open+0x338/0x1150 [lmv] [ 4356.293821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.296086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.299633] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4356.301616] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4356.303716] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.306365] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4356.308440] ? do_raw_spin_unlock+0x75/0x190 [ 4356.310015] ? _raw_spin_unlock+0x12/0x30 [ 4356.311705] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4356.313847] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4356.315994] ll_atomic_open+0x36a/0xd70 [lustre] [ 4356.318919] lookup_open+0x356/0xb40 [ 4356.320786] do_last+0x313/0xfc0 [ 4356.322153] ? nd_jump_root+0xe5/0x160 [ 4356.323333] ? path_init+0x437/0x520 [ 4356.324284] path_openat+0xf7/0x500 [ 4356.325251] ? do_raw_spin_unlock+0x75/0x190 [ 4356.326769] do_filp_open+0x99/0x140 [ 4356.328272] ? getname_flags+0x6e/0x330 [ 4356.329892] ? __check_object_size+0xff/0x256 [ 4356.331728] ? do_raw_spin_unlock+0x75/0x190 [ 4356.333554] ? _raw_spin_unlock+0x12/0x30 [ 4356.334968] do_sys_openat2+0x2b4/0x410 [ 4356.336590] do_sys_open+0x73/0xa0 [ 4356.337920] __x64_sys_openat+0x24/0x30 [ 4356.340004] do_syscall_64+0xc1/0x440 [ 4356.348384] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4356.356977] RIP: 0033:0x7fdc0afbf332 [ 4356.359530] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4356.373455] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4356.382144] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4356.387849] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4356.397037] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4356.405022] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4356.409437] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4364.054772] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4364.069939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4364.108004] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4364.114490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4364.122608] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4364.125579] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4364.146715] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4364.151291] RAX: 0000000000005a27 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4364.160695] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4364.163789] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4364.171377] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4364.176285] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4364.182194] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4364.192376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4364.193832] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4364.203268] Call Trace: [ 4364.203840] [ 4364.204342] ? show_regs.cold.9+0x22/0x2f [ 4364.207956] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4364.214744] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4364.217952] ? watchdog+0x40/0x40 [ 4364.219293] ? hrtimer_interrupt+0x128/0x2c0 [ 4364.221275] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4364.223967] ? apic_timer_interrupt+0xf/0x20 [ 4364.225272] [ 4364.225978] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4364.232594] do_raw_spin_lock+0x177/0x190 [ 4364.233916] _raw_spin_lock+0x21/0x30 [ 4364.242162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4364.249205] ? do_raw_spin_unlock+0x75/0x190 [ 4364.250945] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4364.252826] ? do_wait_intr+0xf0/0xf0 [ 4364.255702] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4364.258969] mdc_close+0x53b/0xd70 [mdc] [ 4364.260915] lmv_close+0x18d/0x490 [lmv] [ 4364.262692] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4364.266830] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4364.278337] ll_md_close+0x274/0x800 [lustre] [ 4364.280082] ll_file_release+0x128/0xc50 [lustre] [ 4364.286425] __fput+0x10d/0x440 [ 4364.293143] ____fput+0x12/0x20 [ 4364.294523] task_work_run+0xc6/0x110 [ 4364.295794] exit_to_usermode_loop+0x1dd/0x1f0 [ 4364.298622] do_syscall_64+0x3d6/0x440 [ 4364.301151] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4364.303363] RIP: 0033:0x7fdc0afbfcd8 [ 4364.308754] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4364.323289] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4364.332996] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4364.337392] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4364.341824] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4364.345597] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4364.348911] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4368.051790] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 4368.058901] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4368.096113] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4368.105436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4368.116257] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4368.123400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4368.132640] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4368.147610] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4368.150604] RAX: 000000000000110d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4368.156517] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4368.162300] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4368.167314] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4368.169733] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4368.174873] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4368.180327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4368.184578] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4368.189565] Call Trace: [ 4368.192500] [ 4368.193840] ? show_regs.cold.9+0x22/0x2f [ 4368.196309] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4368.201881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4368.204425] ? watchdog+0x40/0x40 [ 4368.207302] ? hrtimer_interrupt+0x128/0x2c0 [ 4368.212198] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4368.216379] ? apic_timer_interrupt+0xf/0x20 [ 4368.220343] [ 4368.223223] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4368.233633] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4368.242133] do_raw_spin_lock+0x177/0x190 [ 4368.247975] _raw_spin_lock+0x21/0x30 [ 4368.248869] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4368.250319] process_one_work+0x2c8/0x700 [ 4368.259009] worker_thread+0x296/0x6e0 [ 4368.263990] ? rescuer_thread+0x570/0x570 [ 4368.267075] kthread+0x1d1/0x200 [ 4368.268280] ? set_kthread_struct+0x70/0x70 [ 4368.275527] ret_from_fork+0x1f/0x30 [ 4376.055058] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4376.060039] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4376.110355] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4376.119047] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4376.124656] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4376.134090] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 4376.145218] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4376.153038] RAX: 00000000000040e6 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4376.166781] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4376.176826] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4376.186106] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4376.197832] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4376.204086] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4376.218549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4376.227128] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4376.241688] Call Trace: [ 4376.247397] [ 4376.250307] ? show_regs.cold.9+0x22/0x2f [ 4376.252689] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4376.257348] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4376.260849] ? watchdog+0x40/0x40 [ 4376.264424] ? hrtimer_interrupt+0x128/0x2c0 [ 4376.267734] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4376.269638] ? apic_timer_interrupt+0xf/0x20 [ 4376.278160] [ 4376.280008] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4376.287999] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4376.289937] ? dump_stack+0xf6/0x10e [ 4376.291156] do_raw_spin_lock+0x177/0x190 [ 4376.297088] _raw_spin_lock+0x21/0x30 [ 4376.299914] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4376.306770] obd_mod_free+0x46/0x140 [ptlrpc] [ 4376.310531] mdc_commit_open+0x91/0xb0 [mdc] [ 4376.316090] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4376.322380] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4376.327196] after_reply+0xe2c/0x1780 [ptlrpc] [ 4376.330378] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4376.333888] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4376.342905] ? do_wait_intr+0xf0/0xf0 [ 4376.347912] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4376.354009] kthread+0x1d1/0x200 [ 4376.356428] ? set_kthread_struct+0x70/0x70 [ 4376.365197] ret_from_fork+0x1f/0x30 [ 4378.591217] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2353s! [ 4378.596451] Showing busy workqueues and worker pools: [ 4378.599583] workqueue events: flags=0x0 [ 4378.601415] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4378.604641] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4378.608470] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4378.612179] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4378.617859] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4378.621606] pending: kfree_rcu_monitor [ 4378.623652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4378.625871] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4378.631541] workqueue events_long: flags=0x0 [ 4378.633595] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4378.636502] pending: lu_objects_flush [obdclass] [ 4378.639268] workqueue events_unbound: flags=0x2 [ 4378.641466] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4378.644074] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4378.649436] workqueue events_power_efficient: flags=0x80 [ 4378.651709] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4378.654792] pending: neigh_periodic_work, check_lifetime [ 4378.657635] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4378.661411] pending: do_cache_clean, neigh_periodic_work [ 4378.663812] workqueue mm_percpu_wq: flags=0x8 [ 4378.665715] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4378.668511] pending: lru_add_drain_per_cpu [ 4378.670943] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4378.673952] pending: vmstat_update, lru_add_drain_per_cpu [ 4378.676977] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4378.680345] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4378.683206] workqueue writeback: flags=0x4a [ 4378.685378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4378.688384] pending: wb_workfn, wb_workfn [ 4378.690404] workqueue xprtiod: flags=0xa [ 4378.692399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4378.694915] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4378.697691] workqueue ipv6_addrconf: flags=0x40008 [ 4378.700012] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4378.703261] pending: addrconf_verify_work [ 4378.705474] workqueue knbd0-recv: flags=0x1a [ 4378.706991] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4378.709873] in-flight: 655:recv_work [ 4378.711801] workqueue ptlrpc_pinger: flags=0x2 [ 4378.713422] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4378.716306] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4378.719138] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2353s workers=4 idle: 36396 38 11188 [ 4378.722977] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4369s workers=2 idle: 658 [ 4384.053908] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4384.059140] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4384.084426] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4384.089946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4384.094674] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4384.097857] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4384.107788] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4384.112638] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4384.118671] RDX: 0000000000005b65 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4384.122856] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4384.126218] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4384.129989] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4384.133755] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4384.137787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4384.140903] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4384.144527] Call Trace: [ 4384.145825] [ 4384.146836] ? show_regs.cold.9+0x22/0x2f [ 4384.149194] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4384.152689] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4384.155782] ? watchdog+0x40/0x40 [ 4384.157467] ? hrtimer_interrupt+0x128/0x2c0 [ 4384.158906] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4384.160967] ? apic_timer_interrupt+0xf/0x20 [ 4384.162980] [ 4384.163854] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4384.166101] do_raw_spin_lock+0x177/0x190 [ 4384.168044] _raw_spin_lock+0x21/0x30 [ 4384.169759] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4384.172147] ? do_raw_spin_unlock+0x75/0x190 [ 4384.173821] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4384.176179] ? do_wait_intr+0xf0/0xf0 [ 4384.177797] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4384.180529] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4384.183040] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4384.186399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.189222] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4384.191668] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4384.194017] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4384.196123] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.199044] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4384.202227] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4384.205012] lmv_intent_open+0x338/0x1150 [lmv] [ 4384.207139] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.210760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.214653] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4384.217241] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4384.219625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.223670] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4384.228942] ? do_raw_spin_unlock+0x75/0x190 [ 4384.231233] ? _raw_spin_unlock+0x12/0x30 [ 4384.233185] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4384.236070] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4384.239485] ll_atomic_open+0x36a/0xd70 [lustre] [ 4384.246110] lookup_open+0x356/0xb40 [ 4384.248042] do_last+0x313/0xfc0 [ 4384.249941] ? nd_jump_root+0xe5/0x160 [ 4384.252029] ? path_init+0x437/0x520 [ 4384.254612] path_openat+0xf7/0x500 [ 4384.257244] ? do_raw_spin_unlock+0x75/0x190 [ 4384.259787] do_filp_open+0x99/0x140 [ 4384.262451] ? getname_flags+0x6e/0x330 [ 4384.264427] ? __check_object_size+0xff/0x256 [ 4384.267983] ? do_raw_spin_unlock+0x75/0x190 [ 4384.270024] ? _raw_spin_unlock+0x12/0x30 [ 4384.272244] do_sys_openat2+0x2b4/0x410 [ 4384.274160] do_sys_open+0x73/0xa0 [ 4384.276009] __x64_sys_openat+0x24/0x30 [ 4384.278038] do_syscall_64+0xc1/0x440 [ 4384.280452] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4384.283444] RIP: 0033:0x7fdc0afbf332 [ 4384.285425] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4384.295389] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4384.298198] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4384.301563] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4384.304060] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4384.307770] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4384.311183] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4392.054021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4392.056507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4392.080096] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4392.087718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4392.092485] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4392.094795] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4392.105057] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4392.109224] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4392.118429] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4392.130715] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4392.146343] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4392.154477] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4392.160935] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4392.168256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4392.171919] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4392.179454] Call Trace: [ 4392.181998] [ 4392.182926] ? show_regs.cold.9+0x22/0x2f [ 4392.185417] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4392.188345] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4392.191438] ? watchdog+0x40/0x40 [ 4392.194451] ? hrtimer_interrupt+0x128/0x2c0 [ 4392.197326] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4392.207702] ? apic_timer_interrupt+0xf/0x20 [ 4392.216652] [ 4392.217886] ? native_safe_halt+0x2e/0x50 [ 4392.222335] kvm_wait+0xcd/0xe0 [ 4392.224413] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4392.230349] do_raw_spin_lock+0x177/0x190 [ 4392.239062] _raw_spin_lock+0x21/0x30 [ 4392.244266] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4392.248261] ? do_raw_spin_unlock+0x75/0x190 [ 4392.254971] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4392.259274] ? do_wait_intr+0xf0/0xf0 [ 4392.264337] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4392.266727] mdc_close+0x53b/0xd70 [mdc] [ 4392.272142] lmv_close+0x18d/0x490 [lmv] [ 4392.276009] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4392.279075] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4392.286939] ll_md_close+0x274/0x800 [lustre] [ 4392.289258] ll_file_release+0x128/0xc50 [lustre] [ 4392.292375] __fput+0x10d/0x440 [ 4392.297707] ____fput+0x12/0x20 [ 4392.299440] task_work_run+0xc6/0x110 [ 4392.307834] exit_to_usermode_loop+0x1dd/0x1f0 [ 4392.314390] do_syscall_64+0x3d6/0x440 [ 4392.319304] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4392.324616] RIP: 0033:0x7fdc0afbfcd8 [ 4392.328105] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4392.343181] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4392.346773] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4392.349254] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4392.356016] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4392.362430] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4392.369666] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4396.052816] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 4396.068740] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4396.123080] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4396.129198] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4396.134161] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4396.138983] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4396.143013] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4396.151407] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4396.153748] RAX: 0000000000005010 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4396.156369] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4396.159023] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4396.161226] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4396.163993] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4396.167844] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4396.169951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4396.172373] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4396.175445] Call Trace: [ 4396.176375] [ 4396.177572] ? show_regs.cold.9+0x22/0x2f [ 4396.179496] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4396.181436] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4396.185980] ? watchdog+0x40/0x40 [ 4396.187590] ? hrtimer_interrupt+0x128/0x2c0 [ 4396.191113] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4396.193366] ? apic_timer_interrupt+0xf/0x20 [ 4396.195072] [ 4396.196659] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4396.201166] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4396.205821] do_raw_spin_lock+0x177/0x190 [ 4396.208692] _raw_spin_lock+0x21/0x30 [ 4396.209545] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4396.219153] process_one_work+0x2c8/0x700 [ 4396.221186] worker_thread+0x296/0x6e0 [ 4396.223443] ? rescuer_thread+0x570/0x570 [ 4396.225186] kthread+0x1d1/0x200 [ 4396.229151] ? set_kthread_struct+0x70/0x70 [ 4396.234721] ret_from_fork+0x1f/0x30 [ 4404.048237] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4404.051208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4404.086204] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4404.091973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4404.096544] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4404.098896] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4404.110843] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4404.114730] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4404.122522] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4404.135720] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4404.147503] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4404.155936] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4404.164325] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4404.173435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4404.177205] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4404.182228] Call Trace: [ 4404.183370] [ 4404.184877] ? show_regs.cold.9+0x22/0x2f [ 4404.187174] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4404.189556] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4404.191781] ? watchdog+0x40/0x40 [ 4404.193907] ? hrtimer_interrupt+0x128/0x2c0 [ 4404.196723] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4404.204510] ? apic_timer_interrupt+0xf/0x20 [ 4404.208928] [ 4404.210682] ? native_safe_halt+0x2e/0x50 [ 4404.213838] kvm_wait+0xcd/0xe0 [ 4404.215486] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4404.217919] ? dump_stack+0xf6/0x10e [ 4404.221156] do_raw_spin_lock+0x177/0x190 [ 4404.226880] _raw_spin_lock+0x21/0x30 [ 4404.231943] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4404.239410] obd_mod_free+0x46/0x140 [ptlrpc] [ 4404.241197] mdc_commit_open+0x91/0xb0 [mdc] [ 4404.248504] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4404.251651] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4404.259449] after_reply+0xe2c/0x1780 [ptlrpc] [ 4404.272380] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4404.278792] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4404.285943] ? do_wait_intr+0xf0/0xf0 [ 4404.296906] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4404.300729] kthread+0x1d1/0x200 [ 4404.306977] ? set_kthread_struct+0x70/0x70 [ 4404.310233] ret_from_fork+0x1f/0x30 [ 4408.806484] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2383s! [ 4408.820874] Showing busy workqueues and worker pools: [ 4408.822668] workqueue events: flags=0x0 [ 4408.836022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4408.841793] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4408.851455] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4408.858834] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4408.880028] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4408.886023] pending: kfree_rcu_monitor [ 4408.887556] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4408.905843] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4408.929620] workqueue events_long: flags=0x0 [ 4408.935016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4408.944819] pending: lu_objects_flush [obdclass] [ 4408.959228] workqueue events_unbound: flags=0x2 [ 4408.967847] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4408.974407] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4408.986890] workqueue events_power_efficient: flags=0x80 [ 4408.993087] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4409.009275] pending: neigh_periodic_work, check_lifetime [ 4409.012070] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4409.024202] pending: do_cache_clean, neigh_periodic_work [ 4409.028987] workqueue mm_percpu_wq: flags=0x8 [ 4409.038702] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4409.040324] pending: lru_add_drain_per_cpu [ 4409.053893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4409.069988] pending: vmstat_update, lru_add_drain_per_cpu [ 4409.072151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4409.087118] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4409.090189] workqueue writeback: flags=0x4a [ 4409.092135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4409.095072] pending: wb_workfn, wb_workfn [ 4409.097037] workqueue xprtiod: flags=0xa [ 4409.106022] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4409.109642] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4409.113152] workqueue ipv6_addrconf: flags=0x40008 [ 4409.116938] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4409.124815] pending: addrconf_verify_work [ 4409.128423] workqueue knbd0-recv: flags=0x1a [ 4409.129984] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4409.142278] in-flight: 655:recv_work [ 4409.149970] workqueue ptlrpc_pinger: flags=0x2 [ 4409.151614] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4409.162959] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4409.170377] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2384s workers=4 idle: 36396 38 11188 [ 4409.176274] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4399s workers=2 idle: 658 [ 4412.052056] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 4412.061865] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4412.146589] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4412.167357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4412.196153] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4412.202314] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4412.242894] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4412.245521] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4412.271812] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4412.281250] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4412.304809] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4412.316202] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4412.324885] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4412.338625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4412.344545] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4412.358853] Call Trace: [ 4412.365925] [ 4412.367042] ? show_regs.cold.9+0x22/0x2f [ 4412.379520] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4412.386558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4412.393002] ? watchdog+0x40/0x40 [ 4412.394167] ? hrtimer_interrupt+0x128/0x2c0 [ 4412.398508] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4412.406104] ? apic_timer_interrupt+0xf/0x20 [ 4412.411857] [ 4412.418097] ? native_safe_halt+0x2e/0x50 [ 4412.423655] kvm_wait+0xcd/0xe0 [ 4412.428213] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4412.436501] do_raw_spin_lock+0x177/0x190 [ 4412.438977] _raw_spin_lock+0x21/0x30 [ 4412.446966] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4412.459501] ? do_raw_spin_unlock+0x75/0x190 [ 4412.465572] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4412.477427] ? do_wait_intr+0xf0/0xf0 [ 4412.483321] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4412.493338] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4412.495202] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4412.511274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4412.520050] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4412.525467] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4412.533347] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4412.541448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4412.553019] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4412.561650] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4412.566693] lmv_intent_open+0x338/0x1150 [lmv] [ 4412.572406] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4412.577605] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4412.582374] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4412.591978] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4412.603129] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4412.614608] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4412.618312] ? do_raw_spin_unlock+0x75/0x190 [ 4412.628865] ? _raw_spin_unlock+0x12/0x30 [ 4412.635046] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4412.640530] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4412.646470] ll_atomic_open+0x36a/0xd70 [lustre] [ 4412.647901] lookup_open+0x356/0xb40 [ 4412.659416] do_last+0x313/0xfc0 [ 4412.666381] ? nd_jump_root+0xe5/0x160 [ 4412.681922] ? path_init+0x437/0x520 [ 4412.689361] path_openat+0xf7/0x500 [ 4412.696627] ? do_raw_spin_unlock+0x75/0x190 [ 4412.698205] do_filp_open+0x99/0x140 [ 4412.710048] ? getname_flags+0x6e/0x330 [ 4412.713564] ? __check_object_size+0xff/0x256 [ 4412.727325] ? do_raw_spin_unlock+0x75/0x190 [ 4412.728772] ? _raw_spin_unlock+0x12/0x30 [ 4412.744219] do_sys_openat2+0x2b4/0x410 [ 4412.746540] do_sys_open+0x73/0xa0 [ 4412.757104] __x64_sys_openat+0x24/0x30 [ 4412.758458] do_syscall_64+0xc1/0x440 [ 4412.779918] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4412.781709] RIP: 0033:0x7fdc0afbf332 [ 4412.802396] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4412.828243] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4412.839118] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4412.851434] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4412.866085] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4412.875965] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4412.896009] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4419.400957] rcu: INFO: rcu_sched self-detected stall on CPU [ 4419.404888] rcu: 0-...!: (1417079 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 4419.408945] (t=2400057 jiffies g=270761 q=3303) [ 4419.410955] rcu: rcu_sched kthread starved for 2394989 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 4419.415363] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4419.418833] rcu: RCU grace-period kthread stack dump: [ 4419.421427] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4419.424993] Call Trace: [ 4419.426171] __schedule+0x351/0xcb0 [ 4419.427658] schedule+0xc0/0x180 [ 4419.429318] schedule_timeout+0xb4/0x190 [ 4419.431746] ? do_raw_spin_unlock+0x75/0x190 [ 4419.433613] ? __next_timer_interrupt+0x160/0x160 [ 4419.436799] ? prepare_to_swait_event+0xf3/0x140 [ 4419.438928] rcu_gp_kthread+0x8dd/0x15b0 [ 4419.440820] ? force_qs_rnp+0x2c0/0x2c0 [ 4419.442867] kthread+0x1d1/0x200 [ 4419.451263] ? set_kthread_struct+0x70/0x70 [ 4419.453149] ret_from_fork+0x1f/0x30 [ 4419.458114] rcu: Stack dump where RCU GP kthread last ran: [ 4419.461954] Sending NMI from CPU 0 to CPUs 3: [ 4419.465448] NMI backtrace for cpu 3 [ 4419.465454] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4419.465457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4419.465458] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4419.465460] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4419.465462] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 [ 4419.465465] RAX: 0000000000000417 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4419.465478] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4419.465479] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4419.465480] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4419.465482] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4419.465483] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4419.465484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4419.465485] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4419.465486] Call Trace: [ 4419.465487] [ 4419.465488] ? show_regs.cold.9+0x22/0x2f [ 4419.465489] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4419.465490] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4419.465491] ? nmi_handle+0xbd/0x210 [ 4419.465491] ? default_do_nmi+0x4b/0x1b0 [ 4419.465492] ? do_nmi+0x280/0x380 [ 4419.465493] ? end_repeat_nmi+0x16/0x1f [ 4419.465494] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4419.465495] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4419.465497] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4419.465498] [ 4419.465499] do_raw_spin_lock+0x177/0x190 [ 4419.465500] _raw_spin_lock+0x21/0x30 [ 4419.465501] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4419.465501] ? do_raw_spin_unlock+0x75/0x190 [ 4419.465503] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4419.465504] ? do_wait_intr+0xf0/0xf0 [ 4419.465505] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4419.465506] mdc_close+0x53b/0xd70 [mdc] [ 4419.465507] lmv_close+0x18d/0x490 [lmv] [ 4419.465508] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4419.465509] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4419.465510] ll_md_close+0x274/0x800 [lustre] [ 4419.465511] ll_file_release+0x128/0xc50 [lustre] [ 4419.465512] __fput+0x10d/0x440 [ 4419.465513] ____fput+0x12/0x20 [ 4419.465514] task_work_run+0xc6/0x110 [ 4419.465515] exit_to_usermode_loop+0x1dd/0x1f0 [ 4419.465516] do_syscall_64+0x3d6/0x440 [ 4419.465517] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4419.465518] RIP: 0033:0x7fdc0afbfcd8 [ 4419.465519] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4419.465521] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4419.465523] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4419.465525] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4419.465526] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4419.465527] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4419.465529] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4419.468393] NMI backtrace for cpu 0 [ 4419.679523] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4419.684669] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4419.688225] Call Trace: [ 4419.695410] [ 4419.697879] ? dump_stack+0xbb/0x10e [ 4419.699918] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4419.703004] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4419.707813] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4419.711344] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4419.715310] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4419.717207] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4419.722609] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4419.727717] ? update_process_times+0x6d/0xc0 [ 4419.731748] ? tick_sched_handle.isra.18+0x27/0x90 [ 4419.734865] ? tick_sched_timer+0x53/0xb0 [ 4419.736806] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4419.739587] ? tick_sched_handle.isra.18+0x90/0x90 [ 4419.741418] ? hrtimer_interrupt+0x128/0x2c0 [ 4419.743736] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4419.746005] ? apic_timer_interrupt+0xf/0x20 [ 4419.748283] [ 4419.748970] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4419.751732] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4419.754574] ? dump_stack+0xf6/0x10e [ 4419.756724] ? do_raw_spin_lock+0x177/0x190 [ 4419.758227] ? _raw_spin_lock+0x21/0x30 [ 4419.760008] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4419.761942] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4419.764302] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4419.765960] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4419.768574] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4419.777262] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4419.789007] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4419.796120] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4419.799903] ? do_wait_intr+0xf0/0xf0 [ 4419.806169] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4419.818602] ? kthread+0x1d1/0x200 [ 4419.823453] ? set_kthread_struct+0x70/0x70 [ 4419.831246] ? ret_from_fork+0x1f/0x30 [ 4424.053003] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 4424.056158] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4424.072608] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4424.083215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4424.087256] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4424.089679] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4424.094825] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4424.102773] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4424.104296] RAX: 0000000000000a92 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4424.115912] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4424.120610] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4424.127320] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4424.130600] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4424.134709] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4424.138547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4424.141323] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4424.144330] Call Trace: [ 4424.145527] [ 4424.146496] ? show_regs.cold.9+0x22/0x2f [ 4424.148381] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4424.150544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4424.152624] ? watchdog+0x40/0x40 [ 4424.156261] ? hrtimer_interrupt+0x128/0x2c0 [ 4424.158600] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4424.169604] ? apic_timer_interrupt+0xf/0x20 [ 4424.175164] [ 4424.176495] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4424.181119] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4424.183405] do_raw_spin_lock+0x177/0x190 [ 4424.191328] _raw_spin_lock+0x21/0x30 [ 4424.198519] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4424.203313] process_one_work+0x2c8/0x700 [ 4424.207857] worker_thread+0x296/0x6e0 [ 4424.209339] ? rescuer_thread+0x570/0x570 [ 4424.210846] kthread+0x1d1/0x200 [ 4424.212437] ? set_kthread_struct+0x70/0x70 [ 4424.214185] ret_from_fork+0x1f/0x30 [ 4439.519079] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2414s! [ 4439.526769] Showing busy workqueues and worker pools: [ 4439.534902] workqueue events: flags=0x0 [ 4439.542196] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4439.554454] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4439.566237] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4439.573601] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4439.585457] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4439.590950] pending: kfree_rcu_monitor [ 4439.594156] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4439.605127] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4439.628104] workqueue events_long: flags=0x0 [ 4439.636432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4439.646284] pending: lu_objects_flush [obdclass] [ 4439.654622] workqueue events_unbound: flags=0x2 [ 4439.660258] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4439.670082] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4439.678092] workqueue events_power_efficient: flags=0x80 [ 4439.684161] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4439.688061] pending: neigh_periodic_work, check_lifetime [ 4439.693463] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4439.706525] pending: do_cache_clean, neigh_periodic_work [ 4439.712645] workqueue mm_percpu_wq: flags=0x8 [ 4439.715904] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4439.723968] pending: lru_add_drain_per_cpu [ 4439.728904] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4439.734107] pending: vmstat_update, lru_add_drain_per_cpu [ 4439.739106] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4439.742227] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4439.748132] workqueue writeback: flags=0x4a [ 4439.753370] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4439.755902] pending: wb_workfn, wb_workfn [ 4439.757808] workqueue xprtiod: flags=0xa [ 4439.760161] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4439.770697] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4439.777929] workqueue ipv6_addrconf: flags=0x40008 [ 4439.783493] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4439.787691] pending: addrconf_verify_work [ 4439.792545] workqueue knbd0-recv: flags=0x1a [ 4439.798407] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4439.805454] in-flight: 655:recv_work [ 4439.807448] workqueue ptlrpc_pinger: flags=0x2 [ 4439.812581] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4439.821262] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4439.826925] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2414s workers=4 idle: 36396 38 11188 [ 4439.839772] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4430s workers=2 idle: 658 [ 4440.053099] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 4440.057662] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4440.084607] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4440.089804] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4440.096855] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4440.099641] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4440.108404] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4440.112103] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4440.115157] RDX: 0000000000002f35 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4440.118646] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4440.121575] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4440.124922] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4440.128152] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4440.132164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4440.135991] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4440.139966] Call Trace: [ 4440.141296] [ 4440.142462] ? show_regs.cold.9+0x22/0x2f [ 4440.143923] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4440.146114] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4440.147963] ? watchdog+0x40/0x40 [ 4440.149279] ? hrtimer_interrupt+0x128/0x2c0 [ 4440.151138] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4440.152742] ? apic_timer_interrupt+0xf/0x20 [ 4440.154508] [ 4440.155436] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4440.157871] do_raw_spin_lock+0x177/0x190 [ 4440.159949] _raw_spin_lock+0x21/0x30 [ 4440.162119] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4440.164867] ? do_raw_spin_unlock+0x75/0x190 [ 4440.166367] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4440.168670] ? do_wait_intr+0xf0/0xf0 [ 4440.170637] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4440.173336] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4440.175524] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4440.178467] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4440.180967] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4440.183393] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4440.185400] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4440.187495] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4440.189846] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4440.192558] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4440.194998] lmv_intent_open+0x338/0x1150 [lmv] [ 4440.197825] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4440.200354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4440.202604] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4440.204954] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4440.207527] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4440.209842] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4440.213142] ? do_raw_spin_unlock+0x75/0x190 [ 4440.215033] ? _raw_spin_unlock+0x12/0x30 [ 4440.216792] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4440.218986] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4440.221006] ll_atomic_open+0x36a/0xd70 [lustre] [ 4440.223082] lookup_open+0x356/0xb40 [ 4440.224537] do_last+0x313/0xfc0 [ 4440.226058] ? nd_jump_root+0xe5/0x160 [ 4440.227565] ? path_init+0x437/0x520 [ 4440.229337] path_openat+0xf7/0x500 [ 4440.230785] ? do_raw_spin_unlock+0x75/0x190 [ 4440.233064] do_filp_open+0x99/0x140 [ 4440.234509] ? getname_flags+0x6e/0x330 [ 4440.236191] ? __check_object_size+0xff/0x256 [ 4440.238173] ? do_raw_spin_unlock+0x75/0x190 [ 4440.239885] ? _raw_spin_unlock+0x12/0x30 [ 4440.241801] do_sys_openat2+0x2b4/0x410 [ 4440.243801] do_sys_open+0x73/0xa0 [ 4440.244916] __x64_sys_openat+0x24/0x30 [ 4440.246151] do_syscall_64+0xc1/0x440 [ 4440.247801] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4440.249903] RIP: 0033:0x7fdc0afbf332 [ 4440.251756] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4440.260564] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4440.264490] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4440.267091] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4440.271961] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4440.276164] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4440.282599] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4444.056692] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4444.060486] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4444.098713] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4444.109033] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4444.113281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4444.119472] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4444.134577] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4444.142548] RAX: 0000000000007750 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4444.146860] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4444.153011] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4444.156631] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4444.159687] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4444.163483] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4444.167166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4444.169701] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4444.174193] Call Trace: [ 4444.175083] [ 4444.175808] ? show_regs.cold.9+0x22/0x2f [ 4444.182178] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4444.188493] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4444.191101] ? watchdog+0x40/0x40 [ 4444.192931] ? hrtimer_interrupt+0x128/0x2c0 [ 4444.195549] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4444.198834] ? apic_timer_interrupt+0xf/0x20 [ 4444.201071] [ 4444.203843] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4444.208915] do_raw_spin_lock+0x177/0x190 [ 4444.217184] _raw_spin_lock+0x21/0x30 [ 4444.219450] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4444.232398] ? do_raw_spin_unlock+0x75/0x190 [ 4444.237539] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4444.256110] ? do_wait_intr+0xf0/0xf0 [ 4444.257247] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4444.258694] mdc_close+0x53b/0xd70 [mdc] [ 4444.266383] lmv_close+0x18d/0x490 [lmv] [ 4444.275480] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4444.283480] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4444.291164] ll_md_close+0x274/0x800 [lustre] [ 4444.292773] ll_file_release+0x128/0xc50 [lustre] [ 4444.305183] __fput+0x10d/0x440 [ 4444.310206] ____fput+0x12/0x20 [ 4444.314096] task_work_run+0xc6/0x110 [ 4444.319049] exit_to_usermode_loop+0x1dd/0x1f0 [ 4444.321988] do_syscall_64+0x3d6/0x440 [ 4444.328527] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4444.333719] RIP: 0033:0x7fdc0afbfcd8 [ 4444.334857] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4444.357909] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4444.364277] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4444.370962] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4444.374618] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4444.380234] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4444.391554] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4452.051164] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 4452.055002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4452.086629] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4452.091963] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4452.096579] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4452.099754] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4452.103012] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4452.112596] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4452.117298] RAX: 000000000000232a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4452.121239] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4452.125260] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4452.127785] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4452.131870] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4452.134709] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4452.138416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4452.141099] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4452.145263] Call Trace: [ 4452.146402] [ 4452.147306] ? show_regs.cold.9+0x22/0x2f [ 4452.151282] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4452.155342] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4452.157786] ? watchdog+0x40/0x40 [ 4452.159717] ? hrtimer_interrupt+0x128/0x2c0 [ 4452.161730] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4452.164064] ? apic_timer_interrupt+0xf/0x20 [ 4452.165908] [ 4452.167355] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4452.170004] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4452.172638] do_raw_spin_lock+0x177/0x190 [ 4452.174440] _raw_spin_lock+0x21/0x30 [ 4452.175958] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4452.178652] process_one_work+0x2c8/0x700 [ 4452.180178] worker_thread+0x296/0x6e0 [ 4452.181921] ? rescuer_thread+0x570/0x570 [ 4452.183574] kthread+0x1d1/0x200 [ 4452.185529] ? set_kthread_struct+0x70/0x70 [ 4452.187365] ret_from_fork+0x1f/0x30 [ 4464.048711] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4464.069172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4464.131616] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4464.139030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4464.147370] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4464.152545] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4464.164499] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4464.168635] RAX: 0000000000005814 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4464.171831] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4464.175143] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4464.177966] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4464.186818] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4464.193815] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4464.198367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4464.200453] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4464.202767] Call Trace: [ 4464.203989] [ 4464.204655] ? show_regs.cold.9+0x22/0x2f [ 4464.206015] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4464.207792] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4464.209588] ? watchdog+0x40/0x40 [ 4464.210876] ? hrtimer_interrupt+0x128/0x2c0 [ 4464.212298] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4464.214353] ? apic_timer_interrupt+0xf/0x20 [ 4464.215784] [ 4464.217426] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4464.219551] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4464.221428] ? dump_stack+0xf6/0x10e [ 4464.222792] do_raw_spin_lock+0x177/0x190 [ 4464.224290] _raw_spin_lock+0x21/0x30 [ 4464.225613] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4464.227542] obd_mod_free+0x46/0x140 [ptlrpc] [ 4464.229408] mdc_commit_open+0x91/0xb0 [mdc] [ 4464.231215] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4464.234197] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4464.236497] after_reply+0xe2c/0x1780 [ptlrpc] [ 4464.238665] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4464.240818] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4464.242596] ? do_wait_intr+0xf0/0xf0 [ 4464.244013] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4464.245846] kthread+0x1d1/0x200 [ 4464.246699] ? set_kthread_struct+0x70/0x70 [ 4464.248027] ret_from_fork+0x1f/0x30 [ 4468.052059] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 4468.056654] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4468.095568] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4468.106172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4468.110948] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4468.116441] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4468.130856] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4468.134701] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4468.137921] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4468.141294] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4468.144635] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4468.149654] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4468.154960] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4468.170099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4468.177650] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4468.184086] Call Trace: [ 4468.185516] [ 4468.186196] ? show_regs.cold.9+0x22/0x2f [ 4468.188110] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4468.190900] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4468.195992] ? watchdog+0x40/0x40 [ 4468.198602] ? hrtimer_interrupt+0x128/0x2c0 [ 4468.205902] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4468.210994] ? apic_timer_interrupt+0xf/0x20 [ 4468.212418] [ 4468.213146] ? native_safe_halt+0x2e/0x50 [ 4468.220691] kvm_wait+0xcd/0xe0 [ 4468.223653] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4468.226921] do_raw_spin_lock+0x177/0x190 [ 4468.229624] _raw_spin_lock+0x21/0x30 [ 4468.231860] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4468.237789] ? do_raw_spin_unlock+0x75/0x190 [ 4468.239987] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4468.244745] ? do_wait_intr+0xf0/0xf0 [ 4468.248348] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4468.253426] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4468.262991] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4468.270602] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4468.274659] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4468.276578] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4468.283477] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4468.286094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4468.288159] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4468.291621] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4468.300556] lmv_intent_open+0x338/0x1150 [lmv] [ 4468.305971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4468.312732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4468.317627] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4468.324607] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4468.331365] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4468.334723] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4468.339777] ? do_raw_spin_unlock+0x75/0x190 [ 4468.342187] ? _raw_spin_unlock+0x12/0x30 [ 4468.344865] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4468.352677] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4468.362071] ll_atomic_open+0x36a/0xd70 [lustre] [ 4468.365759] lookup_open+0x356/0xb40 [ 4468.370722] do_last+0x313/0xfc0 [ 4468.377010] ? nd_jump_root+0xe5/0x160 [ 4468.380384] ? path_init+0x437/0x520 [ 4468.381548] path_openat+0xf7/0x500 [ 4468.383774] ? do_raw_spin_unlock+0x75/0x190 [ 4468.387772] do_filp_open+0x99/0x140 [ 4468.391655] ? getname_flags+0x6e/0x330 [ 4468.393626] ? __check_object_size+0xff/0x256 [ 4468.397802] ? do_raw_spin_unlock+0x75/0x190 [ 4468.400994] ? _raw_spin_unlock+0x12/0x30 [ 4468.403353] do_sys_openat2+0x2b4/0x410 [ 4468.405390] do_sys_open+0x73/0xa0 [ 4468.406815] __x64_sys_openat+0x24/0x30 [ 4468.408719] do_syscall_64+0xc1/0x440 [ 4468.411371] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4468.413044] RIP: 0033:0x7fdc0afbf332 [ 4468.415453] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4468.427267] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4468.432843] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4468.435834] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4468.440409] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4468.443270] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4468.445195] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4470.241307] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2445s! [ 4470.256940] Showing busy workqueues and worker pools: [ 4470.261893] workqueue events: flags=0x0 [ 4470.267700] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4470.272453] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4470.278876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4470.282330] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4470.289120] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4470.293599] pending: kfree_rcu_monitor [ 4470.298154] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4470.306860] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4470.320947] workqueue events_long: flags=0x0 [ 4470.322983] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4470.326021] pending: lu_objects_flush [obdclass] [ 4470.328955] workqueue events_unbound: flags=0x2 [ 4470.330590] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4470.337136] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4470.344993] workqueue events_power_efficient: flags=0x80 [ 4470.351788] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4470.355319] pending: neigh_periodic_work, check_lifetime [ 4470.358513] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4470.362423] pending: do_cache_clean, neigh_periodic_work [ 4470.368511] workqueue mm_percpu_wq: flags=0x8 [ 4470.372506] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4470.375216] pending: lru_add_drain_per_cpu [ 4470.377026] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4470.380433] pending: vmstat_update, lru_add_drain_per_cpu [ 4470.382805] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4470.385613] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4470.388444] workqueue writeback: flags=0x4a [ 4470.391631] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4470.394052] pending: wb_workfn, wb_workfn [ 4470.395991] workqueue xprtiod: flags=0xa [ 4470.398035] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4470.400512] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4470.404228] workqueue ipv6_addrconf: flags=0x40008 [ 4470.406098] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4470.408739] pending: addrconf_verify_work [ 4470.410442] workqueue knbd0-recv: flags=0x1a [ 4470.412392] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4470.415026] in-flight: 655:recv_work [ 4470.416899] workqueue ptlrpc_pinger: flags=0x2 [ 4470.418748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4470.421462] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4470.424328] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2445s workers=4 idle: 36396 38 11188 [ 4470.427758] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4460s workers=2 idle: 658 [ 4472.057961] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4472.062046] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4472.111597] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4472.128467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4472.143032] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4472.150110] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4472.185561] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4472.196950] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4472.208434] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4472.217114] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4472.229641] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4472.242990] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4472.261120] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4472.264053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4472.277810] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4472.286674] Call Trace: [ 4472.287976] [ 4472.289068] ? show_regs.cold.9+0x22/0x2f [ 4472.298056] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4472.303506] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4472.311484] ? watchdog+0x40/0x40 [ 4472.316384] ? hrtimer_interrupt+0x128/0x2c0 [ 4472.321595] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4472.325602] ? apic_timer_interrupt+0xf/0x20 [ 4472.329048] [ 4472.330225] ? native_safe_halt+0x2e/0x50 [ 4472.340507] kvm_wait+0xcd/0xe0 [ 4472.341655] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4472.351316] do_raw_spin_lock+0x177/0x190 [ 4472.355261] _raw_spin_lock+0x21/0x30 [ 4472.362925] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4472.365032] ? do_raw_spin_unlock+0x75/0x190 [ 4472.374967] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4472.382914] ? do_wait_intr+0xf0/0xf0 [ 4472.388340] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4472.392033] mdc_close+0x53b/0xd70 [mdc] [ 4472.394028] lmv_close+0x18d/0x490 [lmv] [ 4472.397482] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4472.401072] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4472.403964] ll_md_close+0x274/0x800 [lustre] [ 4472.407040] ll_file_release+0x128/0xc50 [lustre] [ 4472.412753] __fput+0x10d/0x440 [ 4472.414909] ____fput+0x12/0x20 [ 4472.416730] task_work_run+0xc6/0x110 [ 4472.418598] exit_to_usermode_loop+0x1dd/0x1f0 [ 4472.421720] do_syscall_64+0x3d6/0x440 [ 4472.424785] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4472.427722] RIP: 0033:0x7fdc0afbfcd8 [ 4472.431078] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4472.450322] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4472.460619] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4472.472566] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4472.475235] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4472.480365] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4472.485530] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4480.051885] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4480.058843] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4480.101781] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4480.108276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4480.112304] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4480.115470] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4480.118772] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4480.127233] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4480.130815] RAX: 000000000000490d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4480.134654] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4480.139343] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4480.142376] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4480.145677] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4480.148416] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4480.152492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4480.158794] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4480.161846] Call Trace: [ 4480.163079] [ 4480.163891] ? show_regs.cold.9+0x22/0x2f [ 4480.165819] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4480.167521] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4480.170497] ? watchdog+0x40/0x40 [ 4480.171943] ? hrtimer_interrupt+0x128/0x2c0 [ 4480.173707] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4480.175684] ? apic_timer_interrupt+0xf/0x20 [ 4480.177839] [ 4480.178947] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4480.182391] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4480.186532] do_raw_spin_lock+0x177/0x190 [ 4480.188474] _raw_spin_lock+0x21/0x30 [ 4480.190090] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4480.192778] process_one_work+0x2c8/0x700 [ 4480.194561] worker_thread+0x296/0x6e0 [ 4480.196155] ? rescuer_thread+0x570/0x570 [ 4480.197826] kthread+0x1d1/0x200 [ 4480.199266] ? set_kthread_struct+0x70/0x70 [ 4480.200875] ret_from_fork+0x1f/0x30 [ 4496.048027] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 4496.051725] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4496.052064] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 4496.074743] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4496.074757] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4496.074760] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4496.074772] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4496.074775] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4496.074781] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4496.074783] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4496.074785] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4496.074787] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4496.074789] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4496.074792] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4496.074795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4496.074797] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4496.074806] Call Trace: [ 4496.074817] [ 4496.074822] ? show_regs.cold.9+0x22/0x2f [ 4496.074833] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4496.074839] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4496.074843] ? watchdog+0x40/0x40 [ 4496.074847] ? hrtimer_interrupt+0x128/0x2c0 [ 4496.074861] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4496.074868] ? apic_timer_interrupt+0xf/0x20 [ 4496.074873] [ 4496.074876] ? native_safe_halt+0x2e/0x50 [ 4496.074879] kvm_wait+0xcd/0xe0 [ 4496.074886] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4496.074893] ? dump_stack+0xf6/0x10e [ 4496.074899] do_raw_spin_lock+0x177/0x190 [ 4496.074904] _raw_spin_lock+0x21/0x30 [ 4496.074911] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4496.075411] obd_mod_free+0x46/0x140 [ptlrpc] [ 4496.075617] mdc_commit_open+0x91/0xb0 [mdc] [ 4496.075659] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4496.075944] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4496.076221] after_reply+0xe2c/0x1780 [ptlrpc] [ 4496.080186] Modules linked in: [ 4496.085228] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4496.088808] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4496.089080] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4496.089086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4496.089089] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4496.089102] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4496.089106] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4496.089111] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4496.089113] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4496.089115] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4496.089117] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4496.089119] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4496.089122] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4496.089133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4496.089136] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4496.089143] Call Trace: [ 4496.089149] [ 4496.089154] ? show_regs.cold.9+0x22/0x2f [ 4496.089163] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4496.089169] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4496.089173] ? watchdog+0x40/0x40 [ 4496.089177] ? hrtimer_interrupt+0x128/0x2c0 [ 4496.089180] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4496.089188] ? apic_timer_interrupt+0xf/0x20 [ 4496.089193] [ 4496.089196] ? native_safe_halt+0x2e/0x50 [ 4496.089199] kvm_wait+0xcd/0xe0 [ 4496.089207] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4496.089215] do_raw_spin_lock+0x177/0x190 [ 4496.089219] _raw_spin_lock+0x21/0x30 [ 4496.089224] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4496.089574] ? do_raw_spin_unlock+0x75/0x190 [ 4496.089580] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4496.092473] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4496.092761] ? do_wait_intr+0xf0/0xf0 [ 4496.092771] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4496.093171] kthread+0x1d1/0x200 [ 4496.093178] ? set_kthread_struct+0x70/0x70 [ 4496.093182] ret_from_fork+0x1f/0x30 [ 4496.360899] ? do_wait_intr+0xf0/0xf0 [ 4496.367292] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4496.371178] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4496.375521] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4496.383269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4496.387055] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4496.393063] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4496.395672] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4496.398254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4496.403567] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4496.410291] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4496.416182] lmv_intent_open+0x338/0x1150 [lmv] [ 4496.422117] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4496.428850] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4496.432135] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4496.434371] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4496.437141] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4496.439443] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4496.441863] ? do_raw_spin_unlock+0x75/0x190 [ 4496.443895] ? _raw_spin_unlock+0x12/0x30 [ 4496.445703] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4496.448398] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4496.450679] ll_atomic_open+0x36a/0xd70 [lustre] [ 4496.452926] lookup_open+0x356/0xb40 [ 4496.454975] do_last+0x313/0xfc0 [ 4496.456237] ? nd_jump_root+0xe5/0x160 [ 4496.457826] ? path_init+0x437/0x520 [ 4496.459291] path_openat+0xf7/0x500 [ 4496.460687] ? do_raw_spin_unlock+0x75/0x190 [ 4496.462488] do_filp_open+0x99/0x140 [ 4496.464078] ? getname_flags+0x6e/0x330 [ 4496.465803] ? __check_object_size+0xff/0x256 [ 4496.468250] ? do_raw_spin_unlock+0x75/0x190 [ 4496.475300] ? _raw_spin_unlock+0x12/0x30 [ 4496.476683] do_sys_openat2+0x2b4/0x410 [ 4496.478847] do_sys_open+0x73/0xa0 [ 4496.480749] __x64_sys_openat+0x24/0x30 [ 4496.482694] do_syscall_64+0xc1/0x440 [ 4496.485829] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4496.488841] RIP: 0033:0x7fdc0afbf332 [ 4496.491468] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4496.507359] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4496.510767] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4496.522469] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4496.530670] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4496.537600] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4496.541647] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4500.055495] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4500.065544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4500.093832] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4500.101205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4500.110615] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4500.116144] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4500.125213] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4500.130781] RAX: 0000000000003f96 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4500.138614] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4500.145960] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4500.148888] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4500.153198] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4500.158840] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4500.164535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4500.166610] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4500.169857] Call Trace: [ 4500.172886] [ 4500.174969] ? show_regs.cold.9+0x22/0x2f [ 4500.176836] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4500.180041] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4500.185426] ? watchdog+0x40/0x40 [ 4500.188718] ? hrtimer_interrupt+0x128/0x2c0 [ 4500.192457] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4500.193935] ? apic_timer_interrupt+0xf/0x20 [ 4500.195697] [ 4500.196480] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4500.198531] do_raw_spin_lock+0x177/0x190 [ 4500.200039] _raw_spin_lock+0x21/0x30 [ 4500.201472] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4500.203586] ? do_raw_spin_unlock+0x75/0x190 [ 4500.205213] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4500.207259] ? do_wait_intr+0xf0/0xf0 [ 4500.208717] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4500.210882] mdc_close+0x53b/0xd70 [mdc] [ 4500.212260] lmv_close+0x18d/0x490 [lmv] [ 4500.213804] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4500.215924] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4500.217837] ll_md_close+0x274/0x800 [lustre] [ 4500.219574] ll_file_release+0x128/0xc50 [lustre] [ 4500.221639] __fput+0x10d/0x440 [ 4500.222883] ____fput+0x12/0x20 [ 4500.224073] task_work_run+0xc6/0x110 [ 4500.225427] exit_to_usermode_loop+0x1dd/0x1f0 [ 4500.227012] do_syscall_64+0x3d6/0x440 [ 4500.228226] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4500.229868] RIP: 0033:0x7fdc0afbfcd8 [ 4500.231684] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4500.239063] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4500.242036] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4500.244806] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4500.247759] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4500.250384] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4500.253170] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4500.447431] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2475s! [ 4500.454617] Showing busy workqueues and worker pools: [ 4500.457862] workqueue events: flags=0x0 [ 4500.460712] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4500.464617] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4500.470349] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4500.476629] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4500.483553] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4500.487768] pending: kfree_rcu_monitor [ 4500.490820] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4500.496034] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4500.504094] workqueue events_long: flags=0x0 [ 4500.506839] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4500.511774] pending: lu_objects_flush [obdclass] [ 4500.515210] workqueue events_unbound: flags=0x2 [ 4500.518853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4500.522636] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4500.529598] workqueue events_power_efficient: flags=0x80 [ 4500.532538] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4500.536566] pending: neigh_periodic_work, check_lifetime [ 4500.539325] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4500.543351] pending: do_cache_clean, neigh_periodic_work [ 4500.547064] workqueue mm_percpu_wq: flags=0x8 [ 4500.550333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4500.554261] pending: lru_add_drain_per_cpu [ 4500.556556] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4500.560322] pending: vmstat_update, lru_add_drain_per_cpu [ 4500.562441] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4500.567539] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4500.571306] workqueue writeback: flags=0x4a [ 4500.573712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4500.577078] pending: wb_workfn, wb_workfn [ 4500.578801] workqueue xprtiod: flags=0xa [ 4500.580325] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4500.583945] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4500.588737] workqueue ipv6_addrconf: flags=0x40008 [ 4500.591853] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4500.595749] pending: addrconf_verify_work [ 4500.598097] workqueue knbd0-recv: flags=0x1a [ 4500.600813] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4500.604759] in-flight: 655:recv_work [ 4500.607154] workqueue ptlrpc_pinger: flags=0x2 [ 4500.609714] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4500.612846] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4500.616012] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2475s workers=4 idle: 36396 38 11188 [ 4500.619858] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4491s workers=2 idle: 658 [ 4508.051065] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4508.057398] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4508.094049] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4508.099752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4508.111274] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4508.114412] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4508.122656] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4508.136491] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4508.143309] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4508.148606] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4508.158338] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4508.163883] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4508.176600] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4508.178837] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4508.194227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4508.198730] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4508.201103] Call Trace: [ 4508.206490] [ 4508.208060] ? show_regs.cold.9+0x22/0x2f [ 4508.209553] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4508.215996] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4508.223466] ? watchdog+0x40/0x40 [ 4508.225237] ? hrtimer_interrupt+0x128/0x2c0 [ 4508.229862] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4508.233110] ? apic_timer_interrupt+0xf/0x20 [ 4508.236706] [ 4508.242005] ? native_safe_halt+0x2e/0x50 [ 4508.246619] kvm_wait+0xcd/0xe0 [ 4508.249993] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4508.254863] do_raw_spin_lock+0x177/0x190 [ 4508.257288] _raw_spin_lock+0x21/0x30 [ 4508.258943] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4508.262243] process_one_work+0x2c8/0x700 [ 4508.267649] worker_thread+0x296/0x6e0 [ 4508.271334] ? rescuer_thread+0x570/0x570 [ 4508.274853] kthread+0x1d1/0x200 [ 4508.277810] ? set_kthread_struct+0x70/0x70 [ 4508.281940] ret_from_fork+0x1f/0x30 [ 4524.055193] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4524.064246] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4524.099634] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4524.106321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4524.112408] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4524.115691] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4524.131863] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4524.138798] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4524.146170] RDX: 00000000000039a3 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4524.152608] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4524.157990] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4524.169357] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4524.177194] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4524.181469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4524.183665] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4524.187642] Call Trace: [ 4524.188449] [ 4524.191208] ? show_regs.cold.9+0x22/0x2f [ 4524.193738] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4524.195119] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4524.196499] ? watchdog+0x40/0x40 [ 4524.202532] ? hrtimer_interrupt+0x128/0x2c0 [ 4524.206689] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4524.211450] ? apic_timer_interrupt+0xf/0x20 [ 4524.212855] [ 4524.213534] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4524.218552] do_raw_spin_lock+0x177/0x190 [ 4524.220558] _raw_spin_lock+0x21/0x30 [ 4524.222770] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4524.227328] ? do_raw_spin_unlock+0x75/0x190 [ 4524.230072] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4524.234137] ? do_wait_intr+0xf0/0xf0 [ 4524.236465] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4524.242843] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4524.247602] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4524.251564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.255371] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4524.257579] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4524.260131] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4524.262405] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.265600] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4524.267699] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4524.272463] lmv_intent_open+0x338/0x1150 [lmv] [ 4524.275299] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.283023] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.290476] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4524.294449] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4524.299417] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.313036] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4524.318374] ? do_raw_spin_unlock+0x75/0x190 [ 4524.323203] ? _raw_spin_unlock+0x12/0x30 [ 4524.326530] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4524.333381] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4524.336516] ll_atomic_open+0x36a/0xd70 [lustre] [ 4524.339141] lookup_open+0x356/0xb40 [ 4524.341134] do_last+0x313/0xfc0 [ 4524.345287] ? nd_jump_root+0xe5/0x160 [ 4524.351487] ? path_init+0x437/0x520 [ 4524.352725] path_openat+0xf7/0x500 [ 4524.353790] ? do_raw_spin_unlock+0x75/0x190 [ 4524.366716] do_filp_open+0x99/0x140 [ 4524.371841] ? getname_flags+0x6e/0x330 [ 4524.374384] ? __check_object_size+0xff/0x256 [ 4524.376120] ? do_raw_spin_unlock+0x75/0x190 [ 4524.377524] ? _raw_spin_unlock+0x12/0x30 [ 4524.379573] do_sys_openat2+0x2b4/0x410 [ 4524.383086] do_sys_open+0x73/0xa0 [ 4524.393592] __x64_sys_openat+0x24/0x30 [ 4524.394832] do_syscall_64+0xc1/0x440 [ 4524.397751] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4524.401668] RIP: 0033:0x7fdc0afbf332 [ 4524.403806] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4524.419837] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4524.426444] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4524.432158] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4524.435977] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4524.439408] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4524.442548] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4528.049195] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 4528.056870] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4528.058973] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 [ 4528.063434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4528.063532] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4528.063540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4528.063543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4528.063557] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4528.063561] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4528.063577] RAX: 000000000000588a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4528.063579] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4528.063582] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4528.063584] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4528.063586] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4528.063589] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4528.063593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4528.063595] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4528.063603] Call Trace: [ 4528.063608] [ 4528.063614] ? show_regs.cold.9+0x22/0x2f [ 4528.063624] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4528.063630] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4528.063634] ? watchdog+0x40/0x40 [ 4528.063639] ? hrtimer_interrupt+0x128/0x2c0 [ 4528.063642] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4528.063651] ? apic_timer_interrupt+0xf/0x20 [ 4528.063655] [ 4528.063658] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4528.063666] do_raw_spin_lock+0x177/0x190 [ 4528.063670] _raw_spin_lock+0x21/0x30 [ 4528.063675] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4528.075954] dns_resolver [ 4528.104147] ? do_raw_spin_unlock+0x75/0x190 [ 4528.104166] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4528.109766] intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror [ 4528.114599] ? do_wait_intr+0xf0/0xf0 [ 4528.117059] dm_region_hash dm_log dm_mod [ 4528.126171] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4528.129946] sha512_ssse3 sha512_generic [ 4528.129959] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4528.129966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4528.129969] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4528.129980] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4528.129985] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4528.129988] RAX: 0000000000006807 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4528.129991] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4528.129993] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4528.129995] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4528.129998] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4528.130001] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4528.130004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4528.133635] mdc_close+0x53b/0xd70 [mdc] [ 4528.144795] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4528.148584] lmv_close+0x18d/0x490 [lmv] [ 4528.148630] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4528.148748] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4528.149053] ll_md_close+0x274/0x800 [lustre] [ 4528.149142] ll_file_release+0x128/0xc50 [lustre] [ 4528.149242] __fput+0x10d/0x440 [ 4528.149251] ____fput+0x12/0x20 [ 4528.149256] task_work_run+0xc6/0x110 [ 4528.149264] exit_to_usermode_loop+0x1dd/0x1f0 [ 4528.149271] do_syscall_64+0x3d6/0x440 [ 4528.149276] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4528.149286] RIP: 0033:0x7fdc0afbfcd8 [ 4528.149293] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4528.149297] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4528.149303] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4528.149305] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4528.149308] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4528.149310] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4528.149312] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4528.421109] Call Trace: [ 4528.426932] [ 4528.427798] ? show_regs.cold.9+0x22/0x2f [ 4528.432632] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4528.434356] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4528.440838] ? watchdog+0x40/0x40 [ 4528.444090] ? hrtimer_interrupt+0x128/0x2c0 [ 4528.448038] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4528.451945] ? apic_timer_interrupt+0xf/0x20 [ 4528.455448] [ 4528.456277] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4528.457992] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4528.461647] ? dump_stack+0xf6/0x10e [ 4528.464255] do_raw_spin_lock+0x177/0x190 [ 4528.466320] _raw_spin_lock+0x21/0x30 [ 4528.469975] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4528.475732] obd_mod_free+0x46/0x140 [ptlrpc] [ 4528.478754] mdc_commit_open+0x91/0xb0 [mdc] [ 4528.480928] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4528.484990] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4528.488755] after_reply+0xe2c/0x1780 [ptlrpc] [ 4528.493903] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4528.499105] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4528.504971] ? do_wait_intr+0xf0/0xf0 [ 4528.508903] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4528.513756] kthread+0x1d1/0x200 [ 4528.515873] ? set_kthread_struct+0x70/0x70 [ 4528.521768] ret_from_fork+0x1f/0x30 [ 4530.657046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2505s! [ 4530.665462] Showing busy workqueues and worker pools: [ 4530.672557] workqueue events: flags=0x0 [ 4530.674816] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4530.683411] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4530.688452] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4530.692547] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4530.698678] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4530.706665] pending: kfree_rcu_monitor [ 4530.710777] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4530.717924] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4530.727829] workqueue events_long: flags=0x0 [ 4530.730855] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4530.735102] pending: lu_objects_flush [obdclass] [ 4530.738814] workqueue events_unbound: flags=0x2 [ 4530.741977] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4530.748354] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4530.755750] workqueue events_power_efficient: flags=0x80 [ 4530.759587] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4530.763460] pending: neigh_periodic_work, check_lifetime [ 4530.767724] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4530.773450] pending: do_cache_clean, neigh_periodic_work [ 4530.777421] workqueue mm_percpu_wq: flags=0x8 [ 4530.780142] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4530.783858] pending: lru_add_drain_per_cpu [ 4530.786632] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4530.790865] pending: vmstat_update, lru_add_drain_per_cpu [ 4530.794356] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4530.797734] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4530.801599] workqueue writeback: flags=0x4a [ 4530.804182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4530.807633] pending: wb_workfn, wb_workfn [ 4530.809715] workqueue xprtiod: flags=0xa [ 4530.811949] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4530.814849] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4530.818823] workqueue ipv6_addrconf: flags=0x40008 [ 4530.821387] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4530.825767] pending: addrconf_verify_work [ 4530.828314] workqueue knbd0-recv: flags=0x1a [ 4530.830575] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4530.833705] in-flight: 655:recv_work [ 4530.835767] workqueue ptlrpc_pinger: flags=0x2 [ 4530.838084] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4530.841339] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4530.844697] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2505s workers=4 idle: 36396 38 11188 [ 4530.848457] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4521s workers=2 idle: 658 [ 4536.052526] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4536.058920] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4536.100277] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4536.111337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4536.116541] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4536.119629] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4536.125803] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4536.142790] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4536.149009] RAX: 0000000000005cf7 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4536.154235] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4536.159706] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4536.166959] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4536.170014] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4536.180314] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4536.186286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4536.190930] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4536.198387] Call Trace: [ 4536.200625] [ 4536.201563] ? show_regs.cold.9+0x22/0x2f [ 4536.204128] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4536.205760] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4536.207688] ? watchdog+0x40/0x40 [ 4536.209225] ? hrtimer_interrupt+0x128/0x2c0 [ 4536.213399] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4536.216362] ? apic_timer_interrupt+0xf/0x20 [ 4536.220295] [ 4536.221211] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4536.223545] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4536.226376] do_raw_spin_lock+0x177/0x190 [ 4536.228333] _raw_spin_lock+0x21/0x30 [ 4536.229689] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4536.233230] process_one_work+0x2c8/0x700 [ 4536.235228] worker_thread+0x296/0x6e0 [ 4536.236832] ? rescuer_thread+0x570/0x570 [ 4536.238472] kthread+0x1d1/0x200 [ 4536.239672] ? set_kthread_struct+0x70/0x70 [ 4536.241762] ret_from_fork+0x1f/0x30 [ 4552.053895] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4552.058454] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4552.086536] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4552.090431] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4552.093175] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4552.095029] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4552.101280] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4552.103601] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4552.105703] RDX: 0000000000006528 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4552.107900] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4552.110836] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4552.113539] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4552.116603] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4552.120927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4552.124016] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4552.128556] Call Trace: [ 4552.129729] [ 4552.138490] ? show_regs.cold.9+0x22/0x2f [ 4552.148889] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4552.151340] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4552.155374] ? watchdog+0x40/0x40 [ 4552.158837] ? hrtimer_interrupt+0x128/0x2c0 [ 4552.160093] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4552.167950] ? apic_timer_interrupt+0xf/0x20 [ 4552.170164] [ 4552.172584] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4552.174711] do_raw_spin_lock+0x177/0x190 [ 4552.177049] _raw_spin_lock+0x21/0x30 [ 4552.182459] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4552.186724] ? do_raw_spin_unlock+0x75/0x190 [ 4552.191933] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4552.193575] ? do_wait_intr+0xf0/0xf0 [ 4552.201134] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4552.206699] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4552.211609] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4552.219086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4552.224121] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4552.228519] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4552.232266] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4552.237060] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4552.246132] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4552.250933] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4552.254759] lmv_intent_open+0x338/0x1150 [lmv] [ 4552.257379] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4552.260602] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4552.262935] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4552.264618] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4552.268004] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4552.270391] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4552.272571] ? do_raw_spin_unlock+0x75/0x190 [ 4552.274626] ? _raw_spin_unlock+0x12/0x30 [ 4552.276412] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4552.278301] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4552.288738] ll_atomic_open+0x36a/0xd70 [lustre] [ 4552.290904] lookup_open+0x356/0xb40 [ 4552.292235] do_last+0x313/0xfc0 [ 4552.295300] ? nd_jump_root+0xe5/0x160 [ 4552.298593] ? path_init+0x437/0x520 [ 4552.304429] path_openat+0xf7/0x500 [ 4552.310913] ? do_raw_spin_unlock+0x75/0x190 [ 4552.313583] do_filp_open+0x99/0x140 [ 4552.316037] ? getname_flags+0x6e/0x330 [ 4552.317991] ? __check_object_size+0xff/0x256 [ 4552.319902] ? do_raw_spin_unlock+0x75/0x190 [ 4552.330180] ? _raw_spin_unlock+0x12/0x30 [ 4552.331897] do_sys_openat2+0x2b4/0x410 [ 4552.334612] do_sys_open+0x73/0xa0 [ 4552.338913] __x64_sys_openat+0x24/0x30 [ 4552.340764] do_syscall_64+0xc1/0x440 [ 4552.342365] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4552.350513] RIP: 0033:0x7fdc0afbf332 [ 4552.353172] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4552.367222] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4552.375858] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4552.388083] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4552.398105] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4552.406096] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4552.413717] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4556.048063] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 4556.052529] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4556.054028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4556.088979] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4556.088993] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4556.088996] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4556.089009] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4556.089014] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4556.089020] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4556.089022] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4556.089024] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4556.089026] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4556.089029] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4556.089033] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4556.089037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4556.089039] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4556.089048] Call Trace: [ 4556.089053] [ 4556.089059] ? show_regs.cold.9+0x22/0x2f [ 4556.089069] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4556.089087] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4556.089092] ? watchdog+0x40/0x40 [ 4556.089097] ? hrtimer_interrupt+0x128/0x2c0 [ 4556.089101] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4556.089108] ? apic_timer_interrupt+0xf/0x20 [ 4556.089113] [ 4556.089116] ? native_safe_halt+0x2e/0x50 [ 4556.089120] kvm_wait+0xcd/0xe0 [ 4556.089127] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4556.089134] ? dump_stack+0xf6/0x10e [ 4556.089140] do_raw_spin_lock+0x177/0x190 [ 4556.089144] _raw_spin_lock+0x21/0x30 [ 4556.089151] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4556.089434] obd_mod_free+0x46/0x140 [ptlrpc] [ 4556.089637] mdc_commit_open+0x91/0xb0 [mdc] [ 4556.089673] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4556.089870] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4556.091772] after_reply+0xe2c/0x1780 [ptlrpc] [ 4556.092012] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4556.092239] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4556.092448] ? do_wait_intr+0xf0/0xf0 [ 4556.092458] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4556.092657] kthread+0x1d1/0x200 [ 4556.092664] ? set_kthread_struct+0x70/0x70 [ 4556.092807] ret_from_fork+0x1f/0x30 [ 4556.242542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4556.266515] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4556.271310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4556.276636] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4556.278223] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4556.291269] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4556.295205] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4556.299860] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4556.303491] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4556.307444] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4556.310503] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4556.316903] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4556.321624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4556.325438] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4556.329837] Call Trace: [ 4556.331107] [ 4556.331809] ? show_regs.cold.9+0x22/0x2f [ 4556.333942] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4556.337434] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4556.341330] ? watchdog+0x40/0x40 [ 4556.342894] ? hrtimer_interrupt+0x128/0x2c0 [ 4556.346804] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4556.349184] ? apic_timer_interrupt+0xf/0x20 [ 4556.351598] [ 4556.352582] ? native_safe_halt+0x2e/0x50 [ 4556.355219] kvm_wait+0xcd/0xe0 [ 4556.356590] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4556.360186] do_raw_spin_lock+0x177/0x190 [ 4556.362844] _raw_spin_lock+0x21/0x30 [ 4556.364112] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4556.365686] ? do_raw_spin_unlock+0x75/0x190 [ 4556.369792] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4556.378216] ? do_wait_intr+0xf0/0xf0 [ 4556.380555] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4556.383511] mdc_close+0x53b/0xd70 [mdc] [ 4556.386130] lmv_close+0x18d/0x490 [lmv] [ 4556.389517] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4556.392056] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4556.394407] ll_md_close+0x274/0x800 [lustre] [ 4556.396548] ll_file_release+0x128/0xc50 [lustre] [ 4556.399607] __fput+0x10d/0x440 [ 4556.404337] ____fput+0x12/0x20 [ 4556.406269] task_work_run+0xc6/0x110 [ 4556.408264] exit_to_usermode_loop+0x1dd/0x1f0 [ 4556.410246] do_syscall_64+0x3d6/0x440 [ 4556.413139] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4556.415482] RIP: 0033:0x7fdc0afbfcd8 [ 4556.418466] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4556.429481] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4556.434366] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4556.437919] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4556.441378] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4556.447942] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4556.451749] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4560.863043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2535s! [ 4560.866273] Showing busy workqueues and worker pools: [ 4560.868169] workqueue events: flags=0x0 [ 4560.870231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4560.872935] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4560.875555] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4560.878229] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4560.881946] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4560.886820] pending: kfree_rcu_monitor [ 4560.888713] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4560.894494] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4560.903019] workqueue events_long: flags=0x0 [ 4560.905488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4560.912117] pending: lu_objects_flush [obdclass] [ 4560.919298] workqueue events_unbound: flags=0x2 [ 4560.922017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4560.931143] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4560.947310] workqueue events_power_efficient: flags=0x80 [ 4560.955507] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4560.965020] pending: neigh_periodic_work, check_lifetime [ 4560.967965] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4560.973114] pending: do_cache_clean, neigh_periodic_work [ 4560.982411] workqueue mm_percpu_wq: flags=0x8 [ 4560.987118] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4560.991128] pending: lru_add_drain_per_cpu [ 4560.996443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4560.999150] pending: vmstat_update, lru_add_drain_per_cpu [ 4561.000546] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4561.003373] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4561.006307] workqueue writeback: flags=0x4a [ 4561.008230] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4561.010521] pending: wb_workfn, wb_workfn [ 4561.013125] workqueue xprtiod: flags=0xa [ 4561.015086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4561.018145] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4561.025063] workqueue ipv6_addrconf: flags=0x40008 [ 4561.027526] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4561.031019] pending: addrconf_verify_work [ 4561.033527] workqueue knbd0-recv: flags=0x1a [ 4561.035562] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4561.040059] in-flight: 655:recv_work [ 4561.043359] workqueue ptlrpc_pinger: flags=0x2 [ 4561.045293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4561.048093] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4561.050980] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2536s workers=4 idle: 36396 38 11188 [ 4561.056173] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4551s workers=2 idle: 658 [ 4564.051039] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4564.054066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4564.106374] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4564.115434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4564.124981] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4564.127807] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4564.138686] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4564.157663] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4564.165509] RAX: 000000000000660e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4564.174493] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4564.179573] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4564.186585] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4564.191721] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4564.196022] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4564.199134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4564.201801] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4564.209486] Call Trace: [ 4564.211404] [ 4564.214440] ? show_regs.cold.9+0x22/0x2f [ 4564.217264] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4564.221962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4564.224876] ? watchdog+0x40/0x40 [ 4564.227673] ? hrtimer_interrupt+0x128/0x2c0 [ 4564.230434] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4564.235810] ? apic_timer_interrupt+0xf/0x20 [ 4564.240493] [ 4564.241718] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4564.249741] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4564.258617] do_raw_spin_lock+0x177/0x190 [ 4564.262444] _raw_spin_lock+0x21/0x30 [ 4564.267008] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4564.270171] process_one_work+0x2c8/0x700 [ 4564.272712] worker_thread+0x296/0x6e0 [ 4564.274518] ? rescuer_thread+0x570/0x570 [ 4564.278140] kthread+0x1d1/0x200 [ 4564.279733] ? set_kthread_struct+0x70/0x70 [ 4564.282155] ret_from_fork+0x1f/0x30 [ 4580.053529] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4580.064160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4580.109673] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4580.122777] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4580.132166] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4580.136073] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4580.148307] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4580.157440] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4580.161084] RDX: 00000000000004b2 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4580.164908] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4580.170140] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4580.180807] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4580.185384] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4580.209308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4580.211209] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4580.222818] Call Trace: [ 4580.223649] [ 4580.224428] ? show_regs.cold.9+0x22/0x2f [ 4580.226333] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4580.228663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4580.235123] ? watchdog+0x40/0x40 [ 4580.237031] ? hrtimer_interrupt+0x128/0x2c0 [ 4580.238999] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4580.241686] ? apic_timer_interrupt+0xf/0x20 [ 4580.244506] [ 4580.245936] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4580.255139] do_raw_spin_lock+0x177/0x190 [ 4580.256934] _raw_spin_lock+0x21/0x30 [ 4580.258987] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4580.263901] ? do_raw_spin_unlock+0x75/0x190 [ 4580.269032] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4580.281029] ? do_wait_intr+0xf0/0xf0 [ 4580.289761] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4580.299776] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4580.304845] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4580.309604] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4580.313936] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4580.318894] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4580.322228] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4580.328815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4580.330744] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4580.345074] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4580.348369] lmv_intent_open+0x338/0x1150 [lmv] [ 4580.351034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4580.354381] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4580.357114] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4580.360105] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4580.363164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4580.366045] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4580.367996] ? do_raw_spin_unlock+0x75/0x190 [ 4580.369829] ? _raw_spin_unlock+0x12/0x30 [ 4580.373438] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4580.375779] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4580.377508] ll_atomic_open+0x36a/0xd70 [lustre] [ 4580.380491] lookup_open+0x356/0xb40 [ 4580.383011] do_last+0x313/0xfc0 [ 4580.384406] ? nd_jump_root+0xe5/0x160 [ 4580.386091] ? path_init+0x437/0x520 [ 4580.392110] path_openat+0xf7/0x500 [ 4580.393453] ? do_raw_spin_unlock+0x75/0x190 [ 4580.394703] do_filp_open+0x99/0x140 [ 4580.396720] ? getname_flags+0x6e/0x330 [ 4580.398358] ? __check_object_size+0xff/0x256 [ 4580.400014] ? do_raw_spin_unlock+0x75/0x190 [ 4580.401858] ? _raw_spin_unlock+0x12/0x30 [ 4580.403575] do_sys_openat2+0x2b4/0x410 [ 4580.405438] do_sys_open+0x73/0xa0 [ 4580.406730] __x64_sys_openat+0x24/0x30 [ 4580.408408] do_syscall_64+0xc1/0x440 [ 4580.409745] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4580.411599] RIP: 0033:0x7fdc0afbf332 [ 4580.412841] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4580.419098] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4580.426225] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4580.429857] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4580.432857] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4580.435697] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4580.438758] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4584.054019] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4584.057263] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4584.078819] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4584.083657] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4584.087755] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4584.090371] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4584.100849] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4584.104291] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4584.107653] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4584.111406] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4584.115151] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4584.118228] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4584.122152] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4584.126119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4584.129124] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4584.132291] Call Trace: [ 4584.133663] [ 4584.134287] ? show_regs.cold.9+0x22/0x2f [ 4584.136209] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4584.138580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4584.144120] ? watchdog+0x40/0x40 [ 4584.145869] ? hrtimer_interrupt+0x128/0x2c0 [ 4584.148127] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4584.150122] ? apic_timer_interrupt+0xf/0x20 [ 4584.152249] [ 4584.153111] ? native_safe_halt+0x2e/0x50 [ 4584.155289] kvm_wait+0xcd/0xe0 [ 4584.156976] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4584.159827] do_raw_spin_lock+0x177/0x190 [ 4584.161657] _raw_spin_lock+0x21/0x30 [ 4584.163029] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4584.165901] ? do_raw_spin_unlock+0x75/0x190 [ 4584.167782] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4584.170283] ? do_wait_intr+0xf0/0xf0 [ 4584.173124] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4584.176251] mdc_close+0x53b/0xd70 [mdc] [ 4584.178569] lmv_close+0x18d/0x490 [lmv] [ 4584.181041] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4584.186045] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4584.189489] ll_md_close+0x274/0x800 [lustre] [ 4584.192289] ll_file_release+0x128/0xc50 [lustre] [ 4584.195031] __fput+0x10d/0x440 [ 4584.196696] ____fput+0x12/0x20 [ 4584.201730] task_work_run+0xc6/0x110 [ 4584.205959] exit_to_usermode_loop+0x1dd/0x1f0 [ 4584.209018] do_syscall_64+0x3d6/0x440 [ 4584.210350] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4584.213787] RIP: 0033:0x7fdc0afbfcd8 [ 4584.214962] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4584.222525] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4584.225642] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4584.228045] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4584.230977] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4584.233856] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4584.236740] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4588.053210] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4588.057993] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4588.084328] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4588.090020] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4588.097605] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4588.101840] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4588.110585] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4588.113732] RAX: 0000000000005371 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4588.116938] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4588.119449] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4588.122614] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4588.126792] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4588.134265] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4588.139078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4588.141852] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4588.147340] Call Trace: [ 4588.148253] [ 4588.149016] ? show_regs.cold.9+0x22/0x2f [ 4588.151654] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4588.161387] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4588.163443] ? watchdog+0x40/0x40 [ 4588.164890] ? hrtimer_interrupt+0x128/0x2c0 [ 4588.167907] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4588.169989] ? apic_timer_interrupt+0xf/0x20 [ 4588.172261] [ 4588.174504] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4588.178691] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4588.184253] ? dump_stack+0xf6/0x10e [ 4588.187353] do_raw_spin_lock+0x177/0x190 [ 4588.189303] _raw_spin_lock+0x21/0x30 [ 4588.190538] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4588.194241] obd_mod_free+0x46/0x140 [ptlrpc] [ 4588.196498] mdc_commit_open+0x91/0xb0 [mdc] [ 4588.202476] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4588.205850] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4588.209245] after_reply+0xe2c/0x1780 [ptlrpc] [ 4588.213403] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4588.217226] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4588.219748] ? do_wait_intr+0xf0/0xf0 [ 4588.225186] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4588.227007] kthread+0x1d1/0x200 [ 4588.233561] ? set_kthread_struct+0x70/0x70 [ 4588.237678] ret_from_fork+0x1f/0x30 [ 4591.071959] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2566s! [ 4591.082338] Showing busy workqueues and worker pools: [ 4591.085344] workqueue events: flags=0x0 [ 4591.087247] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4591.089455] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4591.092583] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4591.096161] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4591.101541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4591.104841] pending: kfree_rcu_monitor [ 4591.107332] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4591.111618] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4591.118909] workqueue events_long: flags=0x0 [ 4591.121126] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4591.129690] pending: lu_objects_flush [obdclass] [ 4591.137562] workqueue events_unbound: flags=0x2 [ 4591.142570] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4591.146331] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4591.150415] workqueue events_power_efficient: flags=0x80 [ 4591.152545] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4591.155014] pending: neigh_periodic_work, check_lifetime [ 4591.157239] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4591.159485] pending: do_cache_clean, neigh_periodic_work [ 4591.161910] workqueue mm_percpu_wq: flags=0x8 [ 4591.163357] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4591.167220] pending: lru_add_drain_per_cpu [ 4591.171660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4591.175199] pending: vmstat_update, lru_add_drain_per_cpu [ 4591.185216] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4591.188951] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4591.191474] workqueue writeback: flags=0x4a [ 4591.192971] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4591.196999] pending: wb_workfn, wb_workfn [ 4591.203475] workqueue xprtiod: flags=0xa [ 4591.207381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4591.214298] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4591.218731] workqueue ipv6_addrconf: flags=0x40008 [ 4591.220948] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4591.224653] pending: addrconf_verify_work [ 4591.226791] workqueue knbd0-recv: flags=0x1a [ 4591.229039] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4591.232915] in-flight: 655:recv_work [ 4591.236920] workqueue ptlrpc_pinger: flags=0x2 [ 4591.239563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4591.244093] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4591.247109] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2566s workers=4 idle: 36396 38 11188 [ 4591.250808] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4581s workers=2 idle: 658 [ 4592.052465] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4592.056652] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4592.093367] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4592.103918] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4592.110808] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4592.114669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4592.122635] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4592.135843] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4592.146356] RAX: 0000000000007f84 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4592.151248] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4592.156800] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4592.161706] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4592.171016] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4592.176514] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4592.181129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4592.186690] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4592.193739] Call Trace: [ 4592.195886] [ 4592.196888] ? show_regs.cold.9+0x22/0x2f [ 4592.198911] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4592.200364] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4592.202664] ? watchdog+0x40/0x40 [ 4592.204733] ? hrtimer_interrupt+0x128/0x2c0 [ 4592.210769] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4592.212619] ? apic_timer_interrupt+0xf/0x20 [ 4592.214749] [ 4592.217155] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4592.225011] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4592.233634] do_raw_spin_lock+0x177/0x190 [ 4592.235899] _raw_spin_lock+0x21/0x30 [ 4592.238306] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4592.242386] process_one_work+0x2c8/0x700 [ 4592.245204] worker_thread+0x296/0x6e0 [ 4592.247257] ? rescuer_thread+0x570/0x570 [ 4592.254325] kthread+0x1d1/0x200 [ 4592.257162] ? set_kthread_struct+0x70/0x70 [ 4592.259319] ret_from_fork+0x1f/0x30 [ 4599.403100] rcu: INFO: rcu_sched self-detected stall on CPU [ 4599.405867] rcu: 0-...!: (1532840 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 4599.410434] (t=2580060 jiffies g=270761 q=3322) [ 4599.412746] rcu: rcu_sched kthread starved for 2574992 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 4599.417554] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4599.421693] rcu: RCU grace-period kthread stack dump: [ 4599.423615] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4599.426779] Call Trace: [ 4599.427393] __schedule+0x351/0xcb0 [ 4599.428551] schedule+0xc0/0x180 [ 4599.430360] schedule_timeout+0xb4/0x190 [ 4599.432598] ? do_raw_spin_unlock+0x75/0x190 [ 4599.435839] ? __next_timer_interrupt+0x160/0x160 [ 4599.439517] ? prepare_to_swait_event+0xf3/0x140 [ 4599.441134] rcu_gp_kthread+0x8dd/0x15b0 [ 4599.443299] ? force_qs_rnp+0x2c0/0x2c0 [ 4599.444367] kthread+0x1d1/0x200 [ 4599.445480] ? set_kthread_struct+0x70/0x70 [ 4599.446786] ret_from_fork+0x1f/0x30 [ 4599.448193] rcu: Stack dump where RCU GP kthread last ran: [ 4599.449596] Sending NMI from CPU 0 to CPUs 3: [ 4599.451060] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 4599.452013] NMI backtrace for cpu 0 [ 4599.456523] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4599.461267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4599.464276] Call Trace: [ 4599.464965] [ 4599.465564] ? dump_stack+0xbb/0x10e [ 4599.466746] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4599.470402] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4599.472002] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4599.477656] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4599.479826] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4599.483896] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4599.490745] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4599.496583] ? update_process_times+0x6d/0xc0 [ 4599.504726] ? tick_sched_handle.isra.18+0x27/0x90 [ 4599.512297] ? tick_sched_timer+0x53/0xb0 [ 4599.515535] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4599.522534] ? tick_sched_handle.isra.18+0x90/0x90 [ 4599.527479] ? hrtimer_interrupt+0x128/0x2c0 [ 4599.535113] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4599.536788] ? apic_timer_interrupt+0xf/0x20 [ 4599.540651] [ 4599.542689] ? native_safe_halt+0x2e/0x50 [ 4599.546138] ? kvm_wait+0xcd/0xe0 [ 4599.547821] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4599.550367] ? dump_stack+0xf6/0x10e [ 4599.551879] ? do_raw_spin_lock+0x177/0x190 [ 4599.555427] ? _raw_spin_lock+0x21/0x30 [ 4599.562987] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4599.566588] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4599.576143] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4599.580604] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4599.583198] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4599.589436] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4599.594892] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4599.601808] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4599.609447] ? do_wait_intr+0xf0/0xf0 [ 4599.615136] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4599.618301] ? kthread+0x1d1/0x200 [ 4599.621248] ? set_kthread_struct+0x70/0x70 [ 4599.625217] ? ret_from_fork+0x1f/0x30 [ 4608.052109] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4608.057688] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4608.084404] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4608.099753] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4608.104903] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4608.107513] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4608.116503] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4608.119373] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4608.122077] RDX: 0000000000005b1b RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4608.127652] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4608.129473] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4608.132462] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4608.135661] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4608.138819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4608.140424] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4608.143254] Call Trace: [ 4608.144849] [ 4608.145654] ? show_regs.cold.9+0x22/0x2f [ 4608.147111] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4608.149045] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4608.150746] ? watchdog+0x40/0x40 [ 4608.151962] ? hrtimer_interrupt+0x128/0x2c0 [ 4608.153986] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4608.155946] ? apic_timer_interrupt+0xf/0x20 [ 4608.157612] [ 4608.158441] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4608.160700] do_raw_spin_lock+0x177/0x190 [ 4608.162456] _raw_spin_lock+0x21/0x30 [ 4608.164337] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4608.167063] ? do_raw_spin_unlock+0x75/0x190 [ 4608.169076] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4608.173364] ? do_wait_intr+0xf0/0xf0 [ 4608.175460] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4608.177322] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4608.182888] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4608.193458] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4608.196513] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4608.200220] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4608.206955] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4608.212188] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4608.214453] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4608.217475] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4608.219989] lmv_intent_open+0x338/0x1150 [lmv] [ 4608.221436] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4608.223318] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4608.225325] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4608.227179] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4608.229361] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4608.231674] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4608.233895] ? do_raw_spin_unlock+0x75/0x190 [ 4608.235693] ? _raw_spin_unlock+0x12/0x30 [ 4608.237459] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4608.239526] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4608.242527] ll_atomic_open+0x36a/0xd70 [lustre] [ 4608.246267] lookup_open+0x356/0xb40 [ 4608.247426] do_last+0x313/0xfc0 [ 4608.253412] ? nd_jump_root+0xe5/0x160 [ 4608.257090] ? path_init+0x437/0x520 [ 4608.260199] path_openat+0xf7/0x500 [ 4608.265535] ? do_raw_spin_unlock+0x75/0x190 [ 4608.268982] do_filp_open+0x99/0x140 [ 4608.270409] ? getname_flags+0x6e/0x330 [ 4608.271700] ? __check_object_size+0xff/0x256 [ 4608.273922] ? do_raw_spin_unlock+0x75/0x190 [ 4608.276675] ? _raw_spin_unlock+0x12/0x30 [ 4608.282052] do_sys_openat2+0x2b4/0x410 [ 4608.284945] do_sys_open+0x73/0xa0 [ 4608.286213] __x64_sys_openat+0x24/0x30 [ 4608.288213] do_syscall_64+0xc1/0x440 [ 4608.292491] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4608.295509] RIP: 0033:0x7fdc0afbf332 [ 4608.299682] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4608.313483] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4608.317829] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4608.319644] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4608.322900] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4608.328253] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4608.335180] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4620.051332] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4620.055398] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4620.078116] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4620.083326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4620.087662] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4620.091081] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4620.093746] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4620.103070] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4620.106623] RAX: 0000000000000ff8 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4620.109643] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4620.113069] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4620.116531] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4620.119619] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4620.122577] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4620.126249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4620.128411] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4620.131434] Call Trace: [ 4620.132478] [ 4620.133485] ? show_regs.cold.9+0x22/0x2f [ 4620.135653] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4620.137607] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4620.139451] ? watchdog+0x40/0x40 [ 4620.141196] ? hrtimer_interrupt+0x128/0x2c0 [ 4620.143209] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4620.145256] ? apic_timer_interrupt+0xf/0x20 [ 4620.147686] [ 4620.148875] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4620.151151] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4620.153366] do_raw_spin_lock+0x177/0x190 [ 4620.155073] _raw_spin_lock+0x21/0x30 [ 4620.156811] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4620.159502] process_one_work+0x2c8/0x700 [ 4620.161734] worker_thread+0x296/0x6e0 [ 4620.163701] ? rescuer_thread+0x570/0x570 [ 4620.165985] kthread+0x1d1/0x200 [ 4620.167441] ? set_kthread_struct+0x70/0x70 [ 4620.169636] ret_from_fork+0x1f/0x30 [ 4621.279044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2596s! [ 4621.281812] Showing busy workqueues and worker pools: [ 4621.283413] workqueue events: flags=0x0 [ 4621.284823] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4621.287041] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4621.289720] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4621.291863] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4621.295484] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4621.298002] pending: kfree_rcu_monitor [ 4621.299457] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4621.301915] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4621.306783] workqueue events_long: flags=0x0 [ 4621.308208] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4621.310228] pending: lu_objects_flush [obdclass] [ 4621.312287] workqueue events_unbound: flags=0x2 [ 4621.313698] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4621.315689] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4621.319428] workqueue events_power_efficient: flags=0x80 [ 4621.321362] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4621.323795] pending: neigh_periodic_work, check_lifetime [ 4621.325671] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4621.327522] pending: do_cache_clean, neigh_periodic_work [ 4621.329645] workqueue mm_percpu_wq: flags=0x8 [ 4621.330855] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4621.333117] pending: lru_add_drain_per_cpu [ 4621.334615] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4621.336859] pending: vmstat_update, lru_add_drain_per_cpu [ 4621.338694] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4621.340948] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4621.342882] workqueue writeback: flags=0x4a [ 4621.344266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4621.346461] pending: wb_workfn, wb_workfn [ 4621.348342] workqueue xprtiod: flags=0xa [ 4621.349641] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4621.351815] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4621.354809] workqueue ipv6_addrconf: flags=0x40008 [ 4621.356244] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4621.358121] pending: addrconf_verify_work [ 4621.359519] workqueue knbd0-recv: flags=0x1a [ 4621.360842] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4621.362816] in-flight: 655:recv_work [ 4621.364100] workqueue ptlrpc_pinger: flags=0x2 [ 4621.365438] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4621.367581] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4621.369671] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2596s workers=4 idle: 36396 38 11188 [ 4621.372141] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4611s workers=2 idle: 658 [ 4624.054024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4624.061340] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4624.092415] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4624.096324] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4624.099357] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4624.103268] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4624.126892] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4624.129740] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4624.137788] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4624.144190] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4624.147132] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4624.152299] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4624.156694] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4624.161149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4624.164335] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4624.168418] Call Trace: [ 4624.169648] [ 4624.170813] ? show_regs.cold.9+0x22/0x2f [ 4624.173727] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4624.176953] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4624.179339] ? watchdog+0x40/0x40 [ 4624.181567] ? hrtimer_interrupt+0x128/0x2c0 [ 4624.186806] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4624.189202] ? apic_timer_interrupt+0xf/0x20 [ 4624.192709] [ 4624.194173] ? native_safe_halt+0x2e/0x50 [ 4624.196427] kvm_wait+0xcd/0xe0 [ 4624.198774] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4624.204246] do_raw_spin_lock+0x177/0x190 [ 4624.206482] _raw_spin_lock+0x21/0x30 [ 4624.209842] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4624.213354] ? do_raw_spin_unlock+0x75/0x190 [ 4624.216230] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4624.219050] ? do_wait_intr+0xf0/0xf0 [ 4624.221462] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4624.225125] mdc_close+0x53b/0xd70 [mdc] [ 4624.228058] lmv_close+0x18d/0x490 [lmv] [ 4624.232400] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4624.236735] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4624.239815] ll_md_close+0x274/0x800 [lustre] [ 4624.243423] ll_file_release+0x128/0xc50 [lustre] [ 4624.247351] __fput+0x10d/0x440 [ 4624.248906] ____fput+0x12/0x20 [ 4624.251737] task_work_run+0xc6/0x110 [ 4624.254318] exit_to_usermode_loop+0x1dd/0x1f0 [ 4624.258274] do_syscall_64+0x3d6/0x440 [ 4624.260319] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4624.263427] RIP: 0033:0x7fdc0afbfcd8 [ 4624.266915] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4624.280644] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4624.285427] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4624.289769] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4624.293775] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4624.297957] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4624.302107] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4636.061161] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4636.070854] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4636.135576] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4636.146847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4636.151730] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4636.164963] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4636.186083] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4636.190670] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4636.199585] RDX: 0000000000005710 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4636.211638] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4636.213844] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4636.225076] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4636.231377] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4636.236887] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4636.242702] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4636.245102] Call Trace: [ 4636.248690] [ 4636.249710] ? show_regs.cold.9+0x22/0x2f [ 4636.255798] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4636.263947] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4636.268574] ? watchdog+0x40/0x40 [ 4636.272003] ? hrtimer_interrupt+0x128/0x2c0 [ 4636.281942] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4636.288874] ? apic_timer_interrupt+0xf/0x20 [ 4636.291377] [ 4636.294030] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4636.296442] do_raw_spin_lock+0x177/0x190 [ 4636.297645] _raw_spin_lock+0x21/0x30 [ 4636.301258] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4636.312812] ? do_raw_spin_unlock+0x75/0x190 [ 4636.315723] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4636.318421] ? do_wait_intr+0xf0/0xf0 [ 4636.320448] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4636.330111] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4636.338077] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4636.343375] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4636.350894] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4636.355329] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4636.358101] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4636.360390] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4636.365736] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4636.372187] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4636.380084] lmv_intent_open+0x338/0x1150 [lmv] [ 4636.382732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4636.387450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4636.390636] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4636.393155] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4636.396094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4636.398649] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4636.401370] ? do_raw_spin_unlock+0x75/0x190 [ 4636.403069] ? _raw_spin_unlock+0x12/0x30 [ 4636.404806] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4636.406931] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4636.409314] ll_atomic_open+0x36a/0xd70 [lustre] [ 4636.412329] lookup_open+0x356/0xb40 [ 4636.414755] do_last+0x313/0xfc0 [ 4636.415949] ? nd_jump_root+0xe5/0x160 [ 4636.418100] ? path_init+0x437/0x520 [ 4636.419743] path_openat+0xf7/0x500 [ 4636.420949] ? do_raw_spin_unlock+0x75/0x190 [ 4636.423536] do_filp_open+0x99/0x140 [ 4636.425492] ? getname_flags+0x6e/0x330 [ 4636.434010] ? __check_object_size+0xff/0x256 [ 4636.436109] ? do_raw_spin_unlock+0x75/0x190 [ 4636.438333] ? _raw_spin_unlock+0x12/0x30 [ 4636.439978] do_sys_openat2+0x2b4/0x410 [ 4636.445108] do_sys_open+0x73/0xa0 [ 4636.447011] __x64_sys_openat+0x24/0x30 [ 4636.452345] do_syscall_64+0xc1/0x440 [ 4636.454070] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4636.458352] RIP: 0033:0x7fdc0afbf332 [ 4636.459833] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4636.470474] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4636.473962] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4636.477085] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4636.480614] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4636.483527] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4636.487076] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4648.051027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4648.051036] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4648.051047] Modules linked in: [ 4648.055731] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4648.055817] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4648.055824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4648.055831] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4648.061788] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 4648.063907] [ 4648.063914] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4648.063930] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4648.063934] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4648.063939] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4648.063941] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4648.063943] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4648.063945] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4648.063948] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4648.063950] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4648.063954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4648.063956] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4648.063964] Call Trace: [ 4648.063969] [ 4648.063975] ? show_regs.cold.9+0x22/0x2f [ 4648.063984] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4648.063990] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4648.063994] ? watchdog+0x40/0x40 [ 4648.063998] ? hrtimer_interrupt+0x128/0x2c0 [ 4648.064002] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4648.064008] ? apic_timer_interrupt+0xf/0x20 [ 4648.064012] [ 4648.064014] ? native_safe_halt+0x2e/0x50 [ 4648.064072] kvm_wait+0xcd/0xe0 [ 4648.064079] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4648.064090] do_raw_spin_lock+0x177/0x190 [ 4648.064094] _raw_spin_lock+0x21/0x30 [ 4648.064098] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4648.064398] process_one_work+0x2c8/0x700 [ 4648.064403] worker_thread+0x296/0x6e0 [ 4648.064406] ? rescuer_thread+0x570/0x570 [ 4648.064408] kthread+0x1d1/0x200 [ 4648.064411] ? set_kthread_struct+0x70/0x70 [ 4648.064414] ret_from_fork+0x1f/0x30 [ 4648.328394] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4648.358178] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4648.366032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4648.373129] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4648.375364] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4648.387590] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4648.400761] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4648.410321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4648.415700] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4648.423575] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4648.427319] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4648.430725] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4648.438694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4648.445786] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4648.454467] Call Trace: [ 4648.455230] [ 4648.461118] ? show_regs.cold.9+0x22/0x2f [ 4648.467128] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4648.471319] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4648.473342] ? watchdog+0x40/0x40 [ 4648.474939] ? hrtimer_interrupt+0x128/0x2c0 [ 4648.477218] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4648.479945] ? apic_timer_interrupt+0xf/0x20 [ 4648.481747] [ 4648.483048] ? native_safe_halt+0x2e/0x50 [ 4648.484406] kvm_wait+0xcd/0xe0 [ 4648.485375] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4648.488727] ? dump_stack+0xf6/0x10e [ 4648.490498] do_raw_spin_lock+0x177/0x190 [ 4648.492806] _raw_spin_lock+0x21/0x30 [ 4648.494527] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4648.497630] obd_mod_free+0x46/0x140 [ptlrpc] [ 4648.499441] mdc_commit_open+0x91/0xb0 [mdc] [ 4648.502853] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4648.505310] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4648.507682] after_reply+0xe2c/0x1780 [ptlrpc] [ 4648.510933] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4648.513516] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4648.515718] ? do_wait_intr+0xf0/0xf0 [ 4648.517551] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4648.520651] kthread+0x1d1/0x200 [ 4648.521601] ? set_kthread_struct+0x70/0x70 [ 4648.523313] ret_from_fork+0x1f/0x30 [ 4651.488114] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2626s! [ 4651.498842] Showing busy workqueues and worker pools: [ 4651.501789] workqueue events: flags=0x0 [ 4651.507760] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4651.510791] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4651.515683] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4651.519241] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4651.524721] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4651.527732] pending: kfree_rcu_monitor [ 4651.529318] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4651.531563] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4651.542820] workqueue events_long: flags=0x0 [ 4651.544345] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4651.552299] pending: lu_objects_flush [obdclass] [ 4651.554402] workqueue events_unbound: flags=0x2 [ 4651.559741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4651.563056] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4651.567977] workqueue events_power_efficient: flags=0x80 [ 4651.573742] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4651.580743] pending: neigh_periodic_work, check_lifetime [ 4651.585694] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4651.592319] pending: do_cache_clean, neigh_periodic_work [ 4651.602924] workqueue mm_percpu_wq: flags=0x8 [ 4651.608505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4651.612346] pending: lru_add_drain_per_cpu [ 4651.620761] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4651.628445] pending: vmstat_update, lru_add_drain_per_cpu [ 4651.632570] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4651.638605] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4651.642244] workqueue writeback: flags=0x4a [ 4651.643936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4651.650551] pending: wb_workfn, wb_workfn [ 4651.652711] workqueue xprtiod: flags=0xa [ 4651.655313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4651.658103] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4651.668537] workqueue ipv6_addrconf: flags=0x40008 [ 4651.671813] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4651.675358] pending: addrconf_verify_work [ 4651.677306] workqueue knbd0-recv: flags=0x1a [ 4651.679027] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4651.686343] in-flight: 655:recv_work [ 4651.692339] workqueue ptlrpc_pinger: flags=0x2 [ 4651.696882] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4651.702042] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4651.704836] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2626s workers=4 idle: 36396 38 11188 [ 4651.708174] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4642s workers=2 idle: 658 [ 4652.054184] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4652.057838] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4652.097570] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4652.107405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4652.121709] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4652.135569] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4652.160295] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4652.168713] RAX: 0000000000006a86 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4652.171485] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4652.176252] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4652.182771] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4652.186878] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4652.190649] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4652.195353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4652.197255] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4652.201302] Call Trace: [ 4652.202421] [ 4652.203334] ? show_regs.cold.9+0x22/0x2f [ 4652.205549] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4652.207579] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4652.209617] ? watchdog+0x40/0x40 [ 4652.210930] ? hrtimer_interrupt+0x128/0x2c0 [ 4652.216728] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4652.222077] ? apic_timer_interrupt+0xf/0x20 [ 4652.227319] [ 4652.228076] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4652.234133] do_raw_spin_lock+0x177/0x190 [ 4652.241254] _raw_spin_lock+0x21/0x30 [ 4652.243313] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4652.252689] ? do_raw_spin_unlock+0x75/0x190 [ 4652.262197] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4652.274066] ? do_wait_intr+0xf0/0xf0 [ 4652.275311] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4652.280383] mdc_close+0x53b/0xd70 [mdc] [ 4652.284304] lmv_close+0x18d/0x490 [lmv] [ 4652.286576] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4652.290316] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4652.292639] ll_md_close+0x274/0x800 [lustre] [ 4652.294777] ll_file_release+0x128/0xc50 [lustre] [ 4652.296729] __fput+0x10d/0x440 [ 4652.298134] ____fput+0x12/0x20 [ 4652.299612] task_work_run+0xc6/0x110 [ 4652.301180] exit_to_usermode_loop+0x1dd/0x1f0 [ 4652.303380] do_syscall_64+0x3d6/0x440 [ 4652.304944] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4652.307340] RIP: 0033:0x7fdc0afbfcd8 [ 4652.308706] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4652.316344] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4652.319423] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4652.322607] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4652.326261] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4652.342461] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4652.351804] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4664.052658] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4664.056167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4664.103242] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4664.108452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4664.113446] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4664.116601] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4664.125199] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4664.132858] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4664.135741] RDX: 0000000000006473 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4664.143949] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4664.147207] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4664.152734] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4664.154928] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4664.161442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4664.166655] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4664.170087] Call Trace: [ 4664.171204] [ 4664.173690] ? show_regs.cold.9+0x22/0x2f [ 4664.176119] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4664.178421] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4664.181533] ? watchdog+0x40/0x40 [ 4664.188031] ? hrtimer_interrupt+0x128/0x2c0 [ 4664.189919] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4664.192074] ? apic_timer_interrupt+0xf/0x20 [ 4664.195245] [ 4664.195912] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4664.197945] do_raw_spin_lock+0x177/0x190 [ 4664.201670] _raw_spin_lock+0x21/0x30 [ 4664.203274] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4664.207406] ? do_raw_spin_unlock+0x75/0x190 [ 4664.208636] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4664.210554] ? do_wait_intr+0xf0/0xf0 [ 4664.213586] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4664.216117] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4664.221751] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4664.226785] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4664.230935] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4664.233518] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4664.236540] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4664.238632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4664.241735] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4664.245249] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4664.248126] lmv_intent_open+0x338/0x1150 [lmv] [ 4664.250310] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4664.254401] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4664.257990] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4664.259966] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4664.263598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4664.266942] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4664.270133] ? do_raw_spin_unlock+0x75/0x190 [ 4664.273112] ? _raw_spin_unlock+0x12/0x30 [ 4664.275173] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4664.277602] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4664.280673] ll_atomic_open+0x36a/0xd70 [lustre] [ 4664.283950] lookup_open+0x356/0xb40 [ 4664.286596] do_last+0x313/0xfc0 [ 4664.288526] ? nd_jump_root+0xe5/0x160 [ 4664.291166] ? path_init+0x437/0x520 [ 4664.293191] path_openat+0xf7/0x500 [ 4664.295207] ? do_raw_spin_unlock+0x75/0x190 [ 4664.297856] do_filp_open+0x99/0x140 [ 4664.299762] ? getname_flags+0x6e/0x330 [ 4664.302251] ? __check_object_size+0xff/0x256 [ 4664.304680] ? do_raw_spin_unlock+0x75/0x190 [ 4664.307111] ? _raw_spin_unlock+0x12/0x30 [ 4664.308767] do_sys_openat2+0x2b4/0x410 [ 4664.309691] do_sys_open+0x73/0xa0 [ 4664.311376] __x64_sys_openat+0x24/0x30 [ 4664.314214] do_syscall_64+0xc1/0x440 [ 4664.316302] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4664.319980] RIP: 0033:0x7fdc0afbf332 [ 4664.321710] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4664.334496] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4664.339437] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4664.343265] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4664.346664] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4664.351664] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4664.357003] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4676.049604] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4676.055342] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4676.059987] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4676.083478] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4676.083494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4676.083497] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4676.083510] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4676.083514] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4676.083525] RAX: 0000000000005485 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4676.083527] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4676.083529] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4676.083532] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4676.083534] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4676.083537] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4676.083540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4676.083543] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4676.083551] Call Trace: [ 4676.083556] [ 4676.083561] ? show_regs.cold.9+0x22/0x2f [ 4676.083647] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4676.083718] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4676.083723] ? watchdog+0x40/0x40 [ 4676.083727] ? hrtimer_interrupt+0x128/0x2c0 [ 4676.083731] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4676.083740] ? apic_timer_interrupt+0xf/0x20 [ 4676.083745] [ 4676.083749] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4676.083754] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4676.083758] ? dump_stack+0xf6/0x10e [ 4676.083765] do_raw_spin_lock+0x177/0x190 [ 4676.083770] _raw_spin_lock+0x21/0x30 [ 4676.083778] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4676.102953] Modules linked in: [ 4676.110265] obd_mod_free+0x46/0x140 [ptlrpc] [ 4676.123824] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4676.124034] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4676.124047] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4676.124057] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4676.124916] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4676.124930] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4676.124934] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4676.124939] RAX: 0000000000007c84 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4676.124942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4676.124944] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4676.124946] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4676.124948] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4676.124951] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4676.124955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4676.124957] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4676.124965] Call Trace: [ 4676.124970] [ 4676.125041] ? show_regs.cold.9+0x22/0x2f [ 4676.125055] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4676.125062] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4676.125070] ? watchdog+0x40/0x40 [ 4676.125075] ? hrtimer_interrupt+0x128/0x2c0 [ 4676.125079] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4676.125092] ? apic_timer_interrupt+0xf/0x20 [ 4676.125097] [ 4676.125100] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4676.125106] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4676.125111] do_raw_spin_lock+0x177/0x190 [ 4676.125116] _raw_spin_lock+0x21/0x30 [ 4676.125122] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4676.125486] process_one_work+0x2c8/0x700 [ 4676.125497] worker_thread+0x296/0x6e0 [ 4676.125506] ? rescuer_thread+0x570/0x570 [ 4676.125510] kthread+0x1d1/0x200 [ 4676.125518] ? set_kthread_struct+0x70/0x70 [ 4676.125522] ret_from_fork+0x1f/0x30 [ 4676.508703] mdc_commit_open+0x91/0xb0 [mdc] [ 4676.511868] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4676.515600] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4676.519502] after_reply+0xe2c/0x1780 [ptlrpc] [ 4676.526433] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4676.530105] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4676.535300] ? do_wait_intr+0xf0/0xf0 [ 4676.536436] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4676.539264] kthread+0x1d1/0x200 [ 4676.541182] ? set_kthread_struct+0x70/0x70 [ 4676.546226] ret_from_fork+0x1f/0x30 [ 4680.054151] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4680.060606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4680.136445] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4680.150316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4680.163550] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4680.170421] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4680.204626] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4680.210724] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4680.217636] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4680.226851] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4680.236008] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4680.246362] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4680.251916] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4680.264628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4680.273420] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4680.282670] Call Trace: [ 4680.286482] [ 4680.289146] ? show_regs.cold.9+0x22/0x2f [ 4680.294481] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4680.300346] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4680.305719] ? watchdog+0x40/0x40 [ 4680.311507] ? hrtimer_interrupt+0x128/0x2c0 [ 4680.318524] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4680.323626] ? apic_timer_interrupt+0xf/0x20 [ 4680.332472] [ 4680.334891] ? native_safe_halt+0x2e/0x50 [ 4680.341391] kvm_wait+0xcd/0xe0 [ 4680.344216] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4680.353875] do_raw_spin_lock+0x177/0x190 [ 4680.356885] _raw_spin_lock+0x21/0x30 [ 4680.360672] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4680.368617] ? do_raw_spin_unlock+0x75/0x190 [ 4680.378118] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4680.383507] ? do_wait_intr+0xf0/0xf0 [ 4680.385379] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4680.390768] mdc_close+0x53b/0xd70 [mdc] [ 4680.394126] lmv_close+0x18d/0x490 [lmv] [ 4680.397875] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4680.406222] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4680.413662] ll_md_close+0x274/0x800 [lustre] [ 4680.416186] ll_file_release+0x128/0xc50 [lustre] [ 4680.422543] __fput+0x10d/0x440 [ 4680.428315] ____fput+0x12/0x20 [ 4680.432022] task_work_run+0xc6/0x110 [ 4680.439656] exit_to_usermode_loop+0x1dd/0x1f0 [ 4680.447169] do_syscall_64+0x3d6/0x440 [ 4680.449663] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4680.456753] RIP: 0033:0x7fdc0afbfcd8 [ 4680.458808] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4680.489414] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4680.495910] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4680.503540] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4680.510809] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4680.520496] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4680.528456] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4682.207821] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2657s! [ 4682.221394] Showing busy workqueues and worker pools: [ 4682.223651] workqueue events: flags=0x0 [ 4682.225697] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4682.228070] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4682.244444] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4682.252728] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4682.259335] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4682.265327] pending: kfree_rcu_monitor [ 4682.273419] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4682.284660] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4682.305773] workqueue events_long: flags=0x0 [ 4682.309926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4682.327453] pending: lu_objects_flush [obdclass] [ 4682.339353] workqueue events_unbound: flags=0x2 [ 4682.345436] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4682.357129] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4682.377063] workqueue events_power_efficient: flags=0x80 [ 4682.383875] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4682.389198] pending: neigh_periodic_work, check_lifetime [ 4682.395520] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4682.408246] pending: do_cache_clean, neigh_periodic_work [ 4682.418384] workqueue mm_percpu_wq: flags=0x8 [ 4682.424214] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4682.431440] pending: lru_add_drain_per_cpu [ 4682.444851] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4682.459909] pending: vmstat_update, lru_add_drain_per_cpu [ 4682.469363] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4682.481890] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4682.490651] workqueue writeback: flags=0x4a [ 4682.496948] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4682.502716] pending: wb_workfn, wb_workfn [ 4682.514532] workqueue xprtiod: flags=0xa [ 4682.516059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4682.521570] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4682.529305] workqueue ipv6_addrconf: flags=0x40008 [ 4682.538854] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4682.550616] pending: addrconf_verify_work [ 4682.559068] workqueue knbd0-recv: flags=0x1a [ 4682.561890] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4682.565909] in-flight: 655:recv_work [ 4682.571172] workqueue ptlrpc_pinger: flags=0x2 [ 4682.576525] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4682.586410] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4682.595028] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2657s workers=4 idle: 36396 38 11188 [ 4682.602044] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4673s workers=2 idle: 658 [ 4692.054649] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4692.062584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4692.104189] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4692.111088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4692.120097] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4692.127736] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4692.138785] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4692.144423] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4692.151441] RDX: 0000000000004ee7 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4692.161480] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4692.175819] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4692.183680] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4692.190156] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4692.200047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4692.202827] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4692.210246] Call Trace: [ 4692.211813] [ 4692.214816] ? show_regs.cold.9+0x22/0x2f [ 4692.219077] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4692.221861] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4692.224418] ? watchdog+0x40/0x40 [ 4692.228594] ? hrtimer_interrupt+0x128/0x2c0 [ 4692.231531] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4692.235731] ? apic_timer_interrupt+0xf/0x20 [ 4692.243237] [ 4692.244125] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4692.251280] do_raw_spin_lock+0x177/0x190 [ 4692.253045] _raw_spin_lock+0x21/0x30 [ 4692.254331] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4692.260673] ? do_raw_spin_unlock+0x75/0x190 [ 4692.263843] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4692.266133] ? do_wait_intr+0xf0/0xf0 [ 4692.268398] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4692.273035] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4692.277639] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4692.280504] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.282756] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4692.285369] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4692.287048] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4692.289018] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.291224] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4692.294394] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4692.296751] lmv_intent_open+0x338/0x1150 [lmv] [ 4692.298632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.300778] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.303551] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4692.305370] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4692.307355] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.309661] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4692.311894] ? do_raw_spin_unlock+0x75/0x190 [ 4692.314051] ? _raw_spin_unlock+0x12/0x30 [ 4692.315580] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4692.317733] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4692.319770] ll_atomic_open+0x36a/0xd70 [lustre] [ 4692.321661] lookup_open+0x356/0xb40 [ 4692.323071] do_last+0x313/0xfc0 [ 4692.324228] ? nd_jump_root+0xe5/0x160 [ 4692.325668] ? path_init+0x437/0x520 [ 4692.327045] path_openat+0xf7/0x500 [ 4692.328412] ? do_raw_spin_unlock+0x75/0x190 [ 4692.330028] do_filp_open+0x99/0x140 [ 4692.331419] ? getname_flags+0x6e/0x330 [ 4692.332780] ? __check_object_size+0xff/0x256 [ 4692.335089] ? do_raw_spin_unlock+0x75/0x190 [ 4692.336878] ? _raw_spin_unlock+0x12/0x30 [ 4692.339278] do_sys_openat2+0x2b4/0x410 [ 4692.341266] do_sys_open+0x73/0xa0 [ 4692.343378] __x64_sys_openat+0x24/0x30 [ 4692.344590] do_syscall_64+0xc1/0x440 [ 4692.349168] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4692.351636] RIP: 0033:0x7fdc0afbf332 [ 4692.353775] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4692.368937] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4692.372868] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4692.375646] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4692.379043] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4692.381705] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4692.385024] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4704.051029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4704.054548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4704.082752] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4704.087862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4704.092166] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4704.095587] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4704.098348] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4704.109147] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4704.113285] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4704.117168] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4704.120806] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4704.124841] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4704.128938] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4704.132980] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4704.135546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4704.140159] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4704.146034] Call Trace: [ 4704.147172] [ 4704.148429] ? show_regs.cold.9+0x22/0x2f [ 4704.150432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4704.152057] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4704.154037] ? watchdog+0x40/0x40 [ 4704.155531] ? hrtimer_interrupt+0x128/0x2c0 [ 4704.157117] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4704.161354] ? apic_timer_interrupt+0xf/0x20 [ 4704.164230] [ 4704.165072] ? native_safe_halt+0x2e/0x50 [ 4704.167129] kvm_wait+0xcd/0xe0 [ 4704.168572] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4704.170913] do_raw_spin_lock+0x177/0x190 [ 4704.172597] _raw_spin_lock+0x21/0x30 [ 4704.174681] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4704.179327] process_one_work+0x2c8/0x700 [ 4704.181458] worker_thread+0x296/0x6e0 [ 4704.184709] ? rescuer_thread+0x570/0x570 [ 4704.187723] kthread+0x1d1/0x200 [ 4704.189249] ? set_kthread_struct+0x70/0x70 [ 4704.196296] ret_from_fork+0x1f/0x30 [ 4708.048161] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4708.052401] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4708.054233] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4708.076577] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4708.076594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4708.076597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4708.076611] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4708.076615] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4708.076621] RAX: 0000000000005ec1 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4708.076624] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4708.076626] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4708.076628] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4708.076631] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4708.076634] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4708.076637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4708.076639] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4708.076647] Call Trace: [ 4708.076652] [ 4708.076657] ? show_regs.cold.9+0x22/0x2f [ 4708.076666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4708.076672] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4708.076676] ? watchdog+0x40/0x40 [ 4708.076680] ? hrtimer_interrupt+0x128/0x2c0 [ 4708.076684] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4708.076692] ? apic_timer_interrupt+0xf/0x20 [ 4708.076696] [ 4708.076699] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4708.076703] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4708.076707] ? dump_stack+0xf6/0x10e [ 4708.076713] do_raw_spin_lock+0x177/0x190 [ 4708.076717] _raw_spin_lock+0x21/0x30 [ 4708.076725] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4708.077147] obd_mod_free+0x46/0x140 [ptlrpc] [ 4708.077405] mdc_commit_open+0x91/0xb0 [mdc] [ 4708.077463] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4708.077654] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4708.078543] after_reply+0xe2c/0x1780 [ptlrpc] [ 4708.078782] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4708.083986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4708.084081] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4708.084089] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4708.084091] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4708.084104] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4708.084107] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4708.084111] RAX: 0000000000005e88 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4708.084113] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4708.084115] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4708.084117] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4708.084119] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4708.084122] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4708.084124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4708.084126] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4708.084134] Call Trace: [ 4708.084140] [ 4708.084145] ? show_regs.cold.9+0x22/0x2f [ 4708.084154] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4708.084160] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4708.084165] ? watchdog+0x40/0x40 [ 4708.084170] ? hrtimer_interrupt+0x128/0x2c0 [ 4708.084173] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4708.084183] ? apic_timer_interrupt+0xf/0x20 [ 4708.084187] [ 4708.084189] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4708.084196] do_raw_spin_lock+0x177/0x190 [ 4708.084201] _raw_spin_lock+0x21/0x30 [ 4708.084205] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4708.084642] ? do_raw_spin_unlock+0x75/0x190 [ 4708.084648] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4708.084932] ? do_wait_intr+0xf0/0xf0 [ 4708.084939] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4708.085200] mdc_close+0x53b/0xd70 [mdc] [ 4708.085240] lmv_close+0x18d/0x490 [lmv] [ 4708.085276] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4708.085910] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4708.085996] ll_md_close+0x274/0x800 [lustre] [ 4708.092734] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4708.093223] ? do_wait_intr+0xf0/0xf0 [ 4708.093235] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4708.098446] ll_file_release+0x128/0xc50 [lustre] [ 4708.098584] __fput+0x10d/0x440 [ 4708.098594] ____fput+0x12/0x20 [ 4708.098599] task_work_run+0xc6/0x110 [ 4708.098605] exit_to_usermode_loop+0x1dd/0x1f0 [ 4708.098612] do_syscall_64+0x3d6/0x440 [ 4708.098622] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4708.098631] RIP: 0033:0x7fdc0afbfcd8 [ 4708.098640] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4708.098644] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4708.098649] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4708.098651] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4708.098655] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4708.098657] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4708.098659] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4708.350156] kthread+0x1d1/0x200 [ 4708.351605] ? set_kthread_struct+0x70/0x70 [ 4708.353272] ret_from_fork+0x1f/0x30 [ 4712.929042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2688s! [ 4712.935582] Showing busy workqueues and worker pools: [ 4712.941145] workqueue events: flags=0x0 [ 4712.943695] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4712.948789] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4712.956698] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4712.961185] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4712.965601] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4712.971019] pending: kfree_rcu_monitor [ 4712.973908] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4712.977406] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4712.985236] workqueue events_long: flags=0x0 [ 4712.989213] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4713.003182] pending: lu_objects_flush [obdclass] [ 4713.008053] workqueue events_unbound: flags=0x2 [ 4713.010770] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4713.013866] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4713.025230] workqueue events_power_efficient: flags=0x80 [ 4713.030010] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4713.031652] pending: neigh_periodic_work, check_lifetime [ 4713.038035] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4713.045848] pending: do_cache_clean, neigh_periodic_work [ 4713.051528] workqueue mm_percpu_wq: flags=0x8 [ 4713.053740] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4713.056399] pending: lru_add_drain_per_cpu [ 4713.061357] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4713.064419] pending: vmstat_update, lru_add_drain_per_cpu [ 4713.067992] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4713.073436] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4713.078078] workqueue writeback: flags=0x4a [ 4713.079982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4713.082942] pending: wb_workfn, wb_workfn [ 4713.087405] workqueue xprtiod: flags=0xa [ 4713.089490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4713.092551] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4713.098141] workqueue ipv6_addrconf: flags=0x40008 [ 4713.100807] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4713.103766] pending: addrconf_verify_work [ 4713.105758] workqueue knbd0-recv: flags=0x1a [ 4713.107708] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4713.110132] in-flight: 655:recv_work [ 4713.112056] workqueue ptlrpc_pinger: flags=0x2 [ 4713.113830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4713.117419] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4713.124733] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2688s workers=4 idle: 36396 38 11188 [ 4713.127222] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4703s workers=2 idle: 658 [ 4720.052725] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4720.062040] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4720.083582] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4720.087044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4720.090233] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4720.092089] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4720.103126] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4720.108330] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4720.113681] RDX: 00000000000068cb RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4720.120154] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4720.125540] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4720.130199] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4720.132193] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4720.134558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4720.136126] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4720.138187] Call Trace: [ 4720.139043] [ 4720.139612] ? show_regs.cold.9+0x22/0x2f [ 4720.140746] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4720.142495] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4720.144443] ? watchdog+0x40/0x40 [ 4720.145641] ? hrtimer_interrupt+0x128/0x2c0 [ 4720.146914] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4720.148353] ? apic_timer_interrupt+0xf/0x20 [ 4720.149407] [ 4720.149927] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4720.151525] do_raw_spin_lock+0x177/0x190 [ 4720.152960] _raw_spin_lock+0x21/0x30 [ 4720.157347] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4720.161560] ? do_raw_spin_unlock+0x75/0x190 [ 4720.163683] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4720.166630] ? do_wait_intr+0xf0/0xf0 [ 4720.168830] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4720.171502] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4720.174122] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4720.176746] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.179126] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4720.181436] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4720.183467] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4720.185270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.188187] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4720.190621] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4720.192481] lmv_intent_open+0x338/0x1150 [lmv] [ 4720.196240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.198484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.201350] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4720.203225] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4720.206545] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.209274] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4720.211429] ? do_raw_spin_unlock+0x75/0x190 [ 4720.213069] ? _raw_spin_unlock+0x12/0x30 [ 4720.215199] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4720.217400] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4720.219596] ll_atomic_open+0x36a/0xd70 [lustre] [ 4720.221519] lookup_open+0x356/0xb40 [ 4720.224354] do_last+0x313/0xfc0 [ 4720.225840] ? nd_jump_root+0xe5/0x160 [ 4720.227250] ? path_init+0x437/0x520 [ 4720.228144] path_openat+0xf7/0x500 [ 4720.229897] ? do_raw_spin_unlock+0x75/0x190 [ 4720.231779] do_filp_open+0x99/0x140 [ 4720.233557] ? getname_flags+0x6e/0x330 [ 4720.235260] ? __check_object_size+0xff/0x256 [ 4720.237192] ? do_raw_spin_unlock+0x75/0x190 [ 4720.239252] ? _raw_spin_unlock+0x12/0x30 [ 4720.240927] do_sys_openat2+0x2b4/0x410 [ 4720.242594] do_sys_open+0x73/0xa0 [ 4720.243956] __x64_sys_openat+0x24/0x30 [ 4720.245964] do_syscall_64+0xc1/0x440 [ 4720.247742] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4720.250516] RIP: 0033:0x7fdc0afbf332 [ 4720.252038] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4720.259630] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4720.262778] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4720.266382] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4720.269000] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4720.272489] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4720.275909] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4732.051059] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 4732.059772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4732.111678] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4732.123035] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4732.128294] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4732.132744] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4732.134028] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4732.157979] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4732.167697] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4732.174711] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4732.178970] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4732.187157] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4732.199912] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4732.205526] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4732.211119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4732.224237] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4732.231299] Call Trace: [ 4732.233807] [ 4732.235270] ? show_regs.cold.9+0x22/0x2f [ 4732.237553] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4732.242140] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4732.247923] ? watchdog+0x40/0x40 [ 4732.251122] ? hrtimer_interrupt+0x128/0x2c0 [ 4732.258156] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4732.259639] ? apic_timer_interrupt+0xf/0x20 [ 4732.268094] [ 4732.273018] ? native_safe_halt+0x2e/0x50 [ 4732.283254] kvm_wait+0xcd/0xe0 [ 4732.284414] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4732.291671] do_raw_spin_lock+0x177/0x190 [ 4732.299281] _raw_spin_lock+0x21/0x30 [ 4732.303037] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4732.307758] process_one_work+0x2c8/0x700 [ 4732.312908] worker_thread+0x296/0x6e0 [ 4732.318597] ? rescuer_thread+0x570/0x570 [ 4732.322145] kthread+0x1d1/0x200 [ 4732.326360] ? set_kthread_struct+0x70/0x70 [ 4732.331447] ret_from_fork+0x1f/0x30 [ 4736.054078] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4736.059976] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4736.082666] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4736.088987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4736.093961] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4736.096794] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4736.106916] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4736.111113] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4736.114357] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4736.118430] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4736.122817] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4736.126763] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4736.131983] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4736.136112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4736.140226] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4736.143659] Call Trace: [ 4736.144476] [ 4736.145326] ? show_regs.cold.9+0x22/0x2f [ 4736.147304] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4736.152499] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4736.154871] ? watchdog+0x40/0x40 [ 4736.156489] ? hrtimer_interrupt+0x128/0x2c0 [ 4736.159356] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4736.161475] ? apic_timer_interrupt+0xf/0x20 [ 4736.163230] [ 4736.164019] ? native_safe_halt+0x2e/0x50 [ 4736.166955] kvm_wait+0xcd/0xe0 [ 4736.168704] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4736.171827] do_raw_spin_lock+0x177/0x190 [ 4736.174215] _raw_spin_lock+0x21/0x30 [ 4736.176637] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4736.180455] ? do_raw_spin_unlock+0x75/0x190 [ 4736.182341] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4736.184813] ? do_wait_intr+0xf0/0xf0 [ 4736.186079] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4736.190361] mdc_close+0x53b/0xd70 [mdc] [ 4736.192643] lmv_close+0x18d/0x490 [lmv] [ 4736.194831] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4736.198088] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4736.200152] ll_md_close+0x274/0x800 [lustre] [ 4736.202736] ll_file_release+0x128/0xc50 [lustre] [ 4736.205368] __fput+0x10d/0x440 [ 4736.207367] ____fput+0x12/0x20 [ 4736.210208] task_work_run+0xc6/0x110 [ 4736.212207] exit_to_usermode_loop+0x1dd/0x1f0 [ 4736.214358] do_syscall_64+0x3d6/0x440 [ 4736.215489] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4736.217268] RIP: 0033:0x7fdc0afbfcd8 [ 4736.219158] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4736.225754] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4736.228208] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4736.230419] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4736.234089] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4736.236494] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4736.241308] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4740.048712] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4740.052562] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4740.127530] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4740.139799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4740.151300] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4740.159407] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4740.198087] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4740.209890] RAX: 0000000000007e09 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4740.225261] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4740.231712] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4740.239845] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4740.245033] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4740.281761] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4740.295133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4740.306258] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4740.315416] Call Trace: [ 4740.320401] [ 4740.329023] ? show_regs.cold.9+0x22/0x2f [ 4740.340698] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4740.345403] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4740.347093] ? watchdog+0x40/0x40 [ 4740.355459] ? hrtimer_interrupt+0x128/0x2c0 [ 4740.356963] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4740.362293] ? apic_timer_interrupt+0xf/0x20 [ 4740.365817] [ 4740.366984] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4740.376147] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4740.383094] ? dump_stack+0xf6/0x10e [ 4740.385769] do_raw_spin_lock+0x177/0x190 [ 4740.387263] _raw_spin_lock+0x21/0x30 [ 4740.394379] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4740.398409] obd_mod_free+0x46/0x140 [ptlrpc] [ 4740.403958] mdc_commit_open+0x91/0xb0 [mdc] [ 4740.406164] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4740.420214] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4740.426811] after_reply+0xe2c/0x1780 [ptlrpc] [ 4740.432987] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4740.436230] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4740.443369] ? do_wait_intr+0xf0/0xf0 [ 4740.449039] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4740.453092] kthread+0x1d1/0x200 [ 4740.457705] ? set_kthread_struct+0x70/0x70 [ 4740.459303] ret_from_fork+0x1f/0x30 [ 4743.136204] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2718s! [ 4743.149839] Showing busy workqueues and worker pools: [ 4743.153059] workqueue events: flags=0x0 [ 4743.161443] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4743.170320] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4743.175542] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4743.184943] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4743.199400] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4743.204966] pending: kfree_rcu_monitor [ 4743.213863] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4743.219627] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4743.230874] workqueue events_long: flags=0x0 [ 4743.237234] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4743.242262] pending: lu_objects_flush [obdclass] [ 4743.248884] workqueue events_unbound: flags=0x2 [ 4743.253823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4743.258887] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4743.267116] workqueue events_power_efficient: flags=0x80 [ 4743.270560] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4743.279913] pending: neigh_periodic_work, check_lifetime [ 4743.287614] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4743.291951] pending: do_cache_clean, neigh_periodic_work [ 4743.296058] workqueue mm_percpu_wq: flags=0x8 [ 4743.300660] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4743.305381] pending: lru_add_drain_per_cpu [ 4743.307894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4743.316759] pending: vmstat_update, lru_add_drain_per_cpu [ 4743.322522] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4743.329931] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4743.338693] workqueue writeback: flags=0x4a [ 4743.347567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4743.353862] pending: wb_workfn, wb_workfn [ 4743.357250] workqueue xprtiod: flags=0xa [ 4743.358809] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4743.362495] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4743.369107] workqueue ipv6_addrconf: flags=0x40008 [ 4743.372461] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4743.377601] pending: addrconf_verify_work [ 4743.379104] workqueue knbd0-recv: flags=0x1a [ 4743.381429] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4743.384772] in-flight: 655:recv_work [ 4743.389507] workqueue ptlrpc_pinger: flags=0x2 [ 4743.394963] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4743.405918] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4743.411365] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2718s workers=4 idle: 36396 38 11188 [ 4743.418250] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4733s workers=2 idle: 658 [ 4748.052078] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4748.056687] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4748.094499] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4748.114867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4748.129007] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4748.136656] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4748.169127] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4748.184402] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4748.191828] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4748.200234] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4748.215509] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4748.224988] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4748.238064] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4748.247389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4748.254597] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4748.272671] Call Trace: [ 4748.274236] [ 4748.284466] ? show_regs.cold.9+0x22/0x2f [ 4748.288213] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4748.296961] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4748.300998] ? watchdog+0x40/0x40 [ 4748.305430] ? hrtimer_interrupt+0x128/0x2c0 [ 4748.318101] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4748.324386] ? apic_timer_interrupt+0xf/0x20 [ 4748.327558] [ 4748.331492] ? native_safe_halt+0x2e/0x50 [ 4748.338541] kvm_wait+0xcd/0xe0 [ 4748.340585] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4748.345143] do_raw_spin_lock+0x177/0x190 [ 4748.356751] _raw_spin_lock+0x21/0x30 [ 4748.361873] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4748.375108] ? do_raw_spin_unlock+0x75/0x190 [ 4748.382548] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4748.395825] ? do_wait_intr+0xf0/0xf0 [ 4748.401144] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4748.415640] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4748.420470] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4748.425848] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4748.431909] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4748.439265] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4748.443389] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4748.453398] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4748.461185] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4748.465982] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4748.470229] lmv_intent_open+0x338/0x1150 [lmv] [ 4748.475847] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4748.480454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4748.485320] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4748.493383] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4748.498817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4748.504760] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4748.511302] ? do_raw_spin_unlock+0x75/0x190 [ 4748.513264] ? _raw_spin_unlock+0x12/0x30 [ 4748.515291] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4748.517692] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4748.523419] ll_atomic_open+0x36a/0xd70 [lustre] [ 4748.529667] lookup_open+0x356/0xb40 [ 4748.531856] do_last+0x313/0xfc0 [ 4748.533847] ? nd_jump_root+0xe5/0x160 [ 4748.535922] ? path_init+0x437/0x520 [ 4748.537468] path_openat+0xf7/0x500 [ 4748.540108] ? do_raw_spin_unlock+0x75/0x190 [ 4748.547138] do_filp_open+0x99/0x140 [ 4748.548833] ? getname_flags+0x6e/0x330 [ 4748.553971] ? __check_object_size+0xff/0x256 [ 4748.557857] ? do_raw_spin_unlock+0x75/0x190 [ 4748.563203] ? _raw_spin_unlock+0x12/0x30 [ 4748.566769] do_sys_openat2+0x2b4/0x410 [ 4748.568186] do_sys_open+0x73/0xa0 [ 4748.569339] __x64_sys_openat+0x24/0x30 [ 4748.575429] do_syscall_64+0xc1/0x440 [ 4748.585251] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4748.588251] RIP: 0033:0x7fdc0afbf332 [ 4748.590790] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4748.604338] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4748.613255] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4748.623338] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4748.630179] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4748.633926] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4748.640660] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4760.051051] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 4760.057287] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4760.112203] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4760.121034] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4760.130022] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4760.132779] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4760.140493] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4760.161557] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4760.168448] RAX: 0000000000003088 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4760.172761] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4760.181627] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4760.185652] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4760.189516] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4760.195034] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4760.203197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4760.207187] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4760.221466] Call Trace: [ 4760.223516] [ 4760.224735] ? show_regs.cold.9+0x22/0x2f [ 4760.228742] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4760.231771] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4760.236284] ? watchdog+0x40/0x40 [ 4760.239179] ? hrtimer_interrupt+0x128/0x2c0 [ 4760.245810] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4760.248571] ? apic_timer_interrupt+0xf/0x20 [ 4760.253824] [ 4760.255497] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4760.260602] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4760.263395] do_raw_spin_lock+0x177/0x190 [ 4760.266521] _raw_spin_lock+0x21/0x30 [ 4760.268440] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4760.272876] process_one_work+0x2c8/0x700 [ 4760.276619] worker_thread+0x296/0x6e0 [ 4760.279601] ? rescuer_thread+0x570/0x570 [ 4760.282645] kthread+0x1d1/0x200 [ 4760.285746] ? set_kthread_struct+0x70/0x70 [ 4760.291638] ret_from_fork+0x1f/0x30 [ 4764.056152] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4764.064382] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4764.119299] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4764.131410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4764.140484] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4764.149566] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4764.166761] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4764.174526] RAX: 0000000000007a88 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4764.178535] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4764.184443] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4764.188744] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4764.194444] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4764.199171] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4764.206538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4764.214282] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4764.226783] Call Trace: [ 4764.228275] [ 4764.229446] ? show_regs.cold.9+0x22/0x2f [ 4764.232317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4764.236041] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4764.238296] ? watchdog+0x40/0x40 [ 4764.242255] ? hrtimer_interrupt+0x128/0x2c0 [ 4764.248497] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4764.251809] ? apic_timer_interrupt+0xf/0x20 [ 4764.254224] [ 4764.254792] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4764.259334] do_raw_spin_lock+0x177/0x190 [ 4764.263213] _raw_spin_lock+0x21/0x30 [ 4764.266671] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4764.269759] ? do_raw_spin_unlock+0x75/0x190 [ 4764.272576] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4764.280244] ? do_wait_intr+0xf0/0xf0 [ 4764.285321] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4764.290709] mdc_close+0x53b/0xd70 [mdc] [ 4764.295247] lmv_close+0x18d/0x490 [lmv] [ 4764.300526] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4764.306741] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4764.312765] ll_md_close+0x274/0x800 [lustre] [ 4764.315550] ll_file_release+0x128/0xc50 [lustre] [ 4764.321486] __fput+0x10d/0x440 [ 4764.322577] ____fput+0x12/0x20 [ 4764.325230] task_work_run+0xc6/0x110 [ 4764.328714] exit_to_usermode_loop+0x1dd/0x1f0 [ 4764.331649] do_syscall_64+0x3d6/0x440 [ 4764.334946] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4764.338803] RIP: 0033:0x7fdc0afbfcd8 [ 4764.341285] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4764.360368] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4764.369197] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4764.373673] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4764.380269] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4764.388715] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4764.395259] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4768.052282] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4768.068651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4768.109576] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4768.119211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4768.125281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4768.130989] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4768.146509] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4768.151240] RAX: 0000000000005db7 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4768.157607] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4768.167955] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4768.172342] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4768.176281] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4768.180323] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4768.189298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4768.194167] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4768.198775] Call Trace: [ 4768.202904] [ 4768.204742] ? show_regs.cold.9+0x22/0x2f [ 4768.208387] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4768.210871] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4768.213324] ? watchdog+0x40/0x40 [ 4768.215648] ? hrtimer_interrupt+0x128/0x2c0 [ 4768.220733] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4768.228871] ? apic_timer_interrupt+0xf/0x20 [ 4768.231586] [ 4768.232874] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4768.240810] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4768.244083] ? dump_stack+0xf6/0x10e [ 4768.246179] do_raw_spin_lock+0x177/0x190 [ 4768.248679] _raw_spin_lock+0x21/0x30 [ 4768.253861] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4768.259302] obd_mod_free+0x46/0x140 [ptlrpc] [ 4768.263878] mdc_commit_open+0x91/0xb0 [mdc] [ 4768.266603] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4768.271551] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4768.277521] after_reply+0xe2c/0x1780 [ptlrpc] [ 4768.282083] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4768.288519] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4768.290189] ? do_wait_intr+0xf0/0xf0 [ 4768.298636] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4768.304671] kthread+0x1d1/0x200 [ 4768.311981] ? set_kthread_struct+0x70/0x70 [ 4768.315403] ret_from_fork+0x1f/0x30 [ 4773.855126] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2748s! [ 4773.859197] Showing busy workqueues and worker pools: [ 4773.861959] workqueue events: flags=0x0 [ 4773.864485] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4773.868608] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4773.874963] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4773.880234] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4773.888279] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4773.892355] pending: kfree_rcu_monitor [ 4773.894152] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4773.898484] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4773.917435] workqueue events_long: flags=0x0 [ 4773.926071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4773.933810] pending: lu_objects_flush [obdclass] [ 4773.943605] workqueue events_unbound: flags=0x2 [ 4773.952498] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4773.968152] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4773.980400] workqueue events_power_efficient: flags=0x80 [ 4773.983276] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4773.987551] pending: neigh_periodic_work, check_lifetime [ 4773.997491] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4774.003525] pending: do_cache_clean, neigh_periodic_work [ 4774.010507] workqueue mm_percpu_wq: flags=0x8 [ 4774.014418] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4774.021469] pending: lru_add_drain_per_cpu [ 4774.034135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4774.040219] pending: vmstat_update, lru_add_drain_per_cpu [ 4774.049986] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4774.057586] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4774.063669] workqueue writeback: flags=0x4a [ 4774.069475] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4774.073134] pending: wb_workfn, wb_workfn [ 4774.075125] workqueue xprtiod: flags=0xa [ 4774.077278] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4774.081685] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4774.088088] workqueue ipv6_addrconf: flags=0x40008 [ 4774.091224] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4774.094828] pending: addrconf_verify_work [ 4774.097967] workqueue knbd0-recv: flags=0x1a [ 4774.103227] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4774.108131] in-flight: 655:recv_work [ 4774.111046] workqueue ptlrpc_pinger: flags=0x2 [ 4774.112791] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4774.116360] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4774.119062] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2749s workers=4 idle: 36396 38 11188 [ 4774.123374] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4764s workers=2 idle: 658 [ 4776.052067] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 4776.057844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4776.099008] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4776.105647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4776.109508] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4776.112832] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4776.123540] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4776.126325] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4776.132803] RDX: 0000000000002555 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4776.137763] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4776.141550] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4776.145166] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4776.151554] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4776.155279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4776.159759] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4776.165053] Call Trace: [ 4776.167481] [ 4776.167992] ? show_regs.cold.9+0x22/0x2f [ 4776.171459] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4776.175018] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4776.177873] ? watchdog+0x40/0x40 [ 4776.179739] ? hrtimer_interrupt+0x128/0x2c0 [ 4776.182981] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4776.184741] ? apic_timer_interrupt+0xf/0x20 [ 4776.188276] [ 4776.189400] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4776.191985] do_raw_spin_lock+0x177/0x190 [ 4776.194317] _raw_spin_lock+0x21/0x30 [ 4776.195721] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4776.197918] ? do_raw_spin_unlock+0x75/0x190 [ 4776.200628] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4776.203376] ? do_wait_intr+0xf0/0xf0 [ 4776.204813] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4776.207364] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4776.209874] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4776.212437] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4776.214783] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4776.216628] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4776.218366] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4776.220037] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4776.221434] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4776.223985] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4776.226170] lmv_intent_open+0x338/0x1150 [lmv] [ 4776.227997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4776.230125] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4776.231745] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4776.233511] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4776.235423] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4776.237613] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4776.239581] ? do_raw_spin_unlock+0x75/0x190 [ 4776.241165] ? _raw_spin_unlock+0x12/0x30 [ 4776.242689] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4776.244603] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4776.246659] ll_atomic_open+0x36a/0xd70 [lustre] [ 4776.248579] lookup_open+0x356/0xb40 [ 4776.249886] do_last+0x313/0xfc0 [ 4776.251136] ? nd_jump_root+0xe5/0x160 [ 4776.252458] ? path_init+0x437/0x520 [ 4776.253969] path_openat+0xf7/0x500 [ 4776.255253] ? do_raw_spin_unlock+0x75/0x190 [ 4776.256957] do_filp_open+0x99/0x140 [ 4776.258294] ? getname_flags+0x6e/0x330 [ 4776.259882] ? __check_object_size+0xff/0x256 [ 4776.261414] ? do_raw_spin_unlock+0x75/0x190 [ 4776.263119] ? _raw_spin_unlock+0x12/0x30 [ 4776.264590] do_sys_openat2+0x2b4/0x410 [ 4776.266154] do_sys_open+0x73/0xa0 [ 4776.267360] __x64_sys_openat+0x24/0x30 [ 4776.268382] do_syscall_64+0xc1/0x440 [ 4776.269376] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4776.271219] RIP: 0033:0x7fdc0afbf332 [ 4776.272692] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4776.279283] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4776.282052] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4776.284723] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4776.287310] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4776.290044] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4776.292554] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4779.406089] rcu: INFO: rcu_sched self-detected stall on CPU [ 4779.408889] rcu: 0-...!: (1643379 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 4779.418853] (t=2760063 jiffies g=270761 q=3380) [ 4779.421678] rcu: rcu_sched kthread starved for 2754995 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 4779.427698] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4779.432702] rcu: RCU grace-period kthread stack dump: [ 4779.435501] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4779.441210] Call Trace: [ 4779.442586] __schedule+0x351/0xcb0 [ 4779.444448] schedule+0xc0/0x180 [ 4779.445685] schedule_timeout+0xb4/0x190 [ 4779.446991] ? do_raw_spin_unlock+0x75/0x190 [ 4779.449701] ? __next_timer_interrupt+0x160/0x160 [ 4779.454464] ? prepare_to_swait_event+0xf3/0x140 [ 4779.457481] rcu_gp_kthread+0x8dd/0x15b0 [ 4779.465519] ? force_qs_rnp+0x2c0/0x2c0 [ 4779.470190] kthread+0x1d1/0x200 [ 4779.474502] ? set_kthread_struct+0x70/0x70 [ 4779.481179] ret_from_fork+0x1f/0x30 [ 4779.486312] rcu: Stack dump where RCU GP kthread last ran: [ 4779.490916] Sending NMI from CPU 0 to CPUs 1: [ 4779.493596] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 4779.496850] NMI backtrace for cpu 0 [ 4779.499228] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4779.504939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4779.513626] Call Trace: [ 4779.516693] [ 4779.517504] ? dump_stack+0xbb/0x10e [ 4779.519669] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4779.525515] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4779.528132] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4779.530501] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4779.533748] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4779.536422] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4779.538693] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4779.541464] ? update_process_times+0x6d/0xc0 [ 4779.543544] ? tick_sched_handle.isra.18+0x27/0x90 [ 4779.545836] ? tick_sched_timer+0x53/0xb0 [ 4779.547822] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4779.551823] ? tick_sched_handle.isra.18+0x90/0x90 [ 4779.557542] ? hrtimer_interrupt+0x128/0x2c0 [ 4779.560637] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4779.563175] ? apic_timer_interrupt+0xf/0x20 [ 4779.570302] [ 4779.571983] ? native_safe_halt+0x2e/0x50 [ 4779.581210] ? kvm_wait+0xcd/0xe0 [ 4779.585327] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4779.594308] ? dump_stack+0xf6/0x10e [ 4779.599615] ? do_raw_spin_lock+0x177/0x190 [ 4779.605578] ? _raw_spin_lock+0x21/0x30 [ 4779.607004] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4779.615568] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4779.621266] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4779.625458] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4779.632634] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4779.642278] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4779.652270] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4779.658588] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4779.664332] ? do_wait_intr+0xf0/0xf0 [ 4779.671948] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4779.678594] ? kthread+0x1d1/0x200 [ 4779.682154] ? set_kthread_struct+0x70/0x70 [ 4779.685676] ? ret_from_fork+0x1f/0x30 [ 4792.054379] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4792.062852] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4792.093378] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4792.114468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4792.124264] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4792.139147] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4792.161574] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4792.167503] RAX: 0000000000006c6f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4792.178801] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4792.192718] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4792.202381] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4792.211478] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4792.230167] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4792.238693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4792.251785] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4792.261618] Call Trace: [ 4792.263663] [ 4792.269546] ? show_regs.cold.9+0x22/0x2f [ 4792.272844] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4792.288854] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4792.298405] ? watchdog+0x40/0x40 [ 4792.305160] ? hrtimer_interrupt+0x128/0x2c0 [ 4792.310351] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4792.317014] ? apic_timer_interrupt+0xf/0x20 [ 4792.321679] [ 4792.323376] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4792.327743] do_raw_spin_lock+0x177/0x190 [ 4792.330288] _raw_spin_lock+0x21/0x30 [ 4792.334918] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4792.337151] ? do_raw_spin_unlock+0x75/0x190 [ 4792.351744] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4792.360582] ? do_wait_intr+0xf0/0xf0 [ 4792.364040] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4792.369885] mdc_close+0x53b/0xd70 [mdc] [ 4792.378357] lmv_close+0x18d/0x490 [lmv] [ 4792.381410] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4792.384379] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4792.389618] ll_md_close+0x274/0x800 [lustre] [ 4792.392502] ll_file_release+0x128/0xc50 [lustre] [ 4792.395635] __fput+0x10d/0x440 [ 4792.397379] ____fput+0x12/0x20 [ 4792.400735] task_work_run+0xc6/0x110 [ 4792.404779] exit_to_usermode_loop+0x1dd/0x1f0 [ 4792.409403] do_syscall_64+0x3d6/0x440 [ 4792.413716] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4792.420413] RIP: 0033:0x7fdc0afbfcd8 [ 4792.422720] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4792.446740] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4792.459331] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4792.468360] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4792.472025] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4792.478815] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4792.484796] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4804.048860] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 4804.051033] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 4804.052207] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 4804.052223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4804.052413] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4804.052423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4804.052427] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4804.052457] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4804.052461] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4804.052470] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4804.052472] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4804.052478] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4804.052484] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4804.052487] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4804.052490] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4804.052493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4804.052496] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4804.052507] Call Trace: [ 4804.052525] [ 4804.052531] ? show_regs.cold.9+0x22/0x2f [ 4804.052542] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4804.052552] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4804.052556] ? watchdog+0x40/0x40 [ 4804.052561] ? hrtimer_interrupt+0x128/0x2c0 [ 4804.052565] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4804.052576] ? apic_timer_interrupt+0xf/0x20 [ 4804.052581] [ 4804.052584] ? native_safe_halt+0x2e/0x50 [ 4804.052588] kvm_wait+0xcd/0xe0 [ 4804.052601] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4804.052613] do_raw_spin_lock+0x177/0x190 [ 4804.052621] _raw_spin_lock+0x21/0x30 [ 4804.052631] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4804.053402] ? do_raw_spin_unlock+0x75/0x190 [ 4804.053413] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4804.053702] ? do_wait_intr+0xf0/0xf0 [ 4804.053718] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4804.054021] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4804.054574] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4804.054878] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4804.055190] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4804.055232] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4804.055262] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4804.055289] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4804.055375] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4804.055842] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4804.055889] lmv_intent_open+0x338/0x1150 [lmv] [ 4804.056427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4804.056579] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4804.056696] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4804.056923] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4804.057249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4804.057364] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4804.057497] ? do_raw_spin_unlock+0x75/0x190 [ 4804.057590] ? _raw_spin_unlock+0x12/0x30 [ 4804.057602] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4804.057644] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4804.058330] ll_atomic_open+0x36a/0xd70 [lustre] [ 4804.058622] lookup_open+0x356/0xb40 [ 4804.058632] do_last+0x313/0xfc0 [ 4804.058637] ? nd_jump_root+0xe5/0x160 [ 4804.058642] ? path_init+0x437/0x520 [ 4804.058646] path_openat+0xf7/0x500 [ 4804.058651] ? do_raw_spin_unlock+0x75/0x190 [ 4804.058657] do_filp_open+0x99/0x140 [ 4804.058664] ? getname_flags+0x6e/0x330 [ 4804.058669] ? __check_object_size+0xff/0x256 [ 4804.058677] ? do_raw_spin_unlock+0x75/0x190 [ 4804.058681] ? _raw_spin_unlock+0x12/0x30 [ 4804.058686] do_sys_openat2+0x2b4/0x410 [ 4804.058691] do_sys_open+0x73/0xa0 [ 4804.058696] __x64_sys_openat+0x24/0x30 [ 4804.058699] do_syscall_64+0xc1/0x440 [ 4804.058706] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4804.058712] RIP: 0033:0x7fdc0afbf332 [ 4804.058725] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4804.058730] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4804.058805] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4804.058808] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4804.058814] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4804.058817] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4804.058820] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4804.059073] Modules linked in: lustre(O) [ 4804.062596] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 4804.072911] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4804.072995] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4804.073002] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4804.073005] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4804.073019] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4804.073024] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4804.073030] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4804.073033] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4804.073035] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4804.073037] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4804.073039] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4804.073042] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4804.073044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4804.073047] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4804.073055] Call Trace: [ 4804.073060] [ 4804.073066] ? show_regs.cold.9+0x22/0x2f [ 4804.073075] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4804.073082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4804.073086] ? watchdog+0x40/0x40 [ 4804.073091] ? hrtimer_interrupt+0x128/0x2c0 [ 4804.073094] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4804.073102] ? apic_timer_interrupt+0xf/0x20 [ 4804.073106] [ 4804.073109] ? native_safe_halt+0x2e/0x50 [ 4804.073113] kvm_wait+0xcd/0xe0 [ 4804.073288] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4804.073297] ? dump_stack+0xf6/0x10e [ 4804.073308] do_raw_spin_lock+0x177/0x190 [ 4804.073313] _raw_spin_lock+0x21/0x30 [ 4804.073319] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4804.127124] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4804.127166] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4804.127174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4804.127181] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4804.141670] obd_mod_free+0x46/0x140 [ptlrpc] [ 4804.147728] [ 4804.147746] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4804.147764] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4804.147769] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4804.147773] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4804.147776] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4804.147778] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4804.147780] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4804.147782] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4804.147785] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4804.147788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4804.147790] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4804.147798] Call Trace: [ 4804.147803] [ 4804.147810] ? show_regs.cold.9+0x22/0x2f [ 4804.147820] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4804.147826] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4804.147830] ? watchdog+0x40/0x40 [ 4804.147833] ? hrtimer_interrupt+0x128/0x2c0 [ 4804.147837] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4804.147844] ? apic_timer_interrupt+0xf/0x20 [ 4804.147849] [ 4804.147853] ? native_safe_halt+0x2e/0x50 [ 4804.147857] kvm_wait+0xcd/0xe0 [ 4804.147865] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4804.147878] do_raw_spin_lock+0x177/0x190 [ 4804.147882] _raw_spin_lock+0x21/0x30 [ 4804.147887] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4804.148247] process_one_work+0x2c8/0x700 [ 4804.148255] worker_thread+0x296/0x6e0 [ 4804.148259] ? rescuer_thread+0x570/0x570 [ 4804.148263] kthread+0x1d1/0x200 [ 4804.148269] ? set_kthread_struct+0x70/0x70 [ 4804.148273] ret_from_fork+0x1f/0x30 [ 4804.876007] mdc_commit_open+0x91/0xb0 [mdc] [ 4804.877951] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4804.880197] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4804.882346] after_reply+0xe2c/0x1780 [ptlrpc] [ 4804.884898] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4804.888950] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4804.891753] ? do_wait_intr+0xf0/0xf0 [ 4804.893261] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4804.895979] kthread+0x1d1/0x200 [ 4804.897747] ? set_kthread_struct+0x70/0x70 [ 4804.898965] ret_from_fork+0x1f/0x30 [ 4804.900035] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2779s! [ 4804.902928] Showing busy workqueues and worker pools: [ 4804.906159] workqueue events: flags=0x0 [ 4804.908355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4804.912052] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4804.917908] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4804.920997] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4804.925612] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4804.929332] pending: kfree_rcu_monitor [ 4804.930899] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4804.933279] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4804.940045] workqueue events_long: flags=0x0 [ 4804.943014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4804.946766] pending: lu_objects_flush [obdclass] [ 4804.950022] workqueue events_unbound: flags=0x2 [ 4804.951454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4804.955760] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4804.958518] workqueue events_power_efficient: flags=0x80 [ 4804.961437] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4804.963410] pending: neigh_periodic_work, check_lifetime [ 4804.966126] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4804.969043] pending: do_cache_clean, neigh_periodic_work [ 4804.973150] workqueue mm_percpu_wq: flags=0x8 [ 4804.976096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4804.979784] pending: lru_add_drain_per_cpu [ 4804.982439] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4804.985992] pending: vmstat_update, lru_add_drain_per_cpu [ 4804.988694] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4804.991621] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4804.994930] workqueue writeback: flags=0x4a [ 4804.996564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4804.998269] pending: wb_workfn, wb_workfn [ 4804.999630] workqueue xprtiod: flags=0xa [ 4805.002761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4805.006252] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4805.011740] workqueue ipv6_addrconf: flags=0x40008 [ 4805.013221] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4805.015496] pending: addrconf_verify_work [ 4805.017900] workqueue knbd0-recv: flags=0x1a [ 4805.020509] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4805.025414] in-flight: 655:recv_work [ 4805.027512] workqueue ptlrpc_pinger: flags=0x2 [ 4805.029124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4805.033229] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4805.035997] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2780s workers=4 idle: 36396 38 11188 [ 4805.040455] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4795s workers=2 idle: 658 [ 4820.061537] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4820.066937] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4820.119933] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4820.137984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4820.149548] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4820.154728] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4820.173397] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4820.185653] RAX: 000000000000678e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4820.188965] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4820.194851] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4820.201703] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4820.209750] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4820.225724] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4820.242554] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4820.246686] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4820.257602] Call Trace: [ 4820.263225] [ 4820.269583] ? show_regs.cold.9+0x22/0x2f [ 4820.275831] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4820.284976] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4820.293945] ? watchdog+0x40/0x40 [ 4820.298023] ? hrtimer_interrupt+0x128/0x2c0 [ 4820.299441] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4820.302030] ? apic_timer_interrupt+0xf/0x20 [ 4820.305030] [ 4820.308043] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4820.314985] do_raw_spin_lock+0x177/0x190 [ 4820.316424] _raw_spin_lock+0x21/0x30 [ 4820.319971] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4820.324967] ? do_raw_spin_unlock+0x75/0x190 [ 4820.329796] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4820.334427] ? do_wait_intr+0xf0/0xf0 [ 4820.338001] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4820.342814] mdc_close+0x53b/0xd70 [mdc] [ 4820.344201] lmv_close+0x18d/0x490 [lmv] [ 4820.347691] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4820.357123] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4820.365474] ll_md_close+0x274/0x800 [lustre] [ 4820.374271] ll_file_release+0x128/0xc50 [lustre] [ 4820.385457] __fput+0x10d/0x440 [ 4820.387907] ____fput+0x12/0x20 [ 4820.396994] task_work_run+0xc6/0x110 [ 4820.400928] exit_to_usermode_loop+0x1dd/0x1f0 [ 4820.402426] do_syscall_64+0x3d6/0x440 [ 4820.416149] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4820.424267] RIP: 0033:0x7fdc0afbfcd8 [ 4820.430791] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4820.460159] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4820.472800] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4820.480962] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4820.485825] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4820.489254] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4820.492924] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4832.050197] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 4832.051157] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 4832.053405] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 4832.053416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4832.053502] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4832.053508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4832.053512] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4832.053527] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4832.053532] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4832.053536] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4832.053539] RDX: 0000000000001b8b RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4832.053553] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4832.053555] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4832.053557] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4832.053560] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4832.053563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4832.053566] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4832.053574] Call Trace: [ 4832.053581] [ 4832.053586] ? show_regs.cold.9+0x22/0x2f [ 4832.053597] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4832.053604] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4832.053608] ? watchdog+0x40/0x40 [ 4832.053613] ? hrtimer_interrupt+0x128/0x2c0 [ 4832.053617] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4832.053625] ? apic_timer_interrupt+0xf/0x20 [ 4832.053630] [ 4832.053634] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4832.053640] do_raw_spin_lock+0x177/0x190 [ 4832.053645] _raw_spin_lock+0x21/0x30 [ 4832.053651] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4832.053949] ? do_raw_spin_unlock+0x75/0x190 [ 4832.053956] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4832.054156] ? do_wait_intr+0xf0/0xf0 [ 4832.054165] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4832.054382] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4832.054606] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4832.054832] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4832.054950] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4832.054989] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4832.055020] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4832.055050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4832.055140] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4832.055374] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4832.055401] lmv_intent_open+0x338/0x1150 [lmv] [ 4832.055431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4832.055516] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4832.055598] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4832.055621] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4832.055709] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4832.055794] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4832.055886] ? do_raw_spin_unlock+0x75/0x190 [ 4832.055893] ? _raw_spin_unlock+0x12/0x30 [ 4832.055899] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4832.055929] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4832.055951] ll_atomic_open+0x36a/0xd70 [lustre] [ 4832.056043] lookup_open+0x356/0xb40 [ 4832.056053] do_last+0x313/0xfc0 [ 4832.056057] ? nd_jump_root+0xe5/0x160 [ 4832.056062] ? path_init+0x437/0x520 [ 4832.056067] path_openat+0xf7/0x500 [ 4832.056073] ? do_raw_spin_unlock+0x75/0x190 [ 4832.056077] do_filp_open+0x99/0x140 [ 4832.056083] ? getname_flags+0x6e/0x330 [ 4832.056087] ? __check_object_size+0xff/0x256 [ 4832.056093] ? do_raw_spin_unlock+0x75/0x190 [ 4832.056096] ? _raw_spin_unlock+0x12/0x30 [ 4832.056100] do_sys_openat2+0x2b4/0x410 [ 4832.056104] do_sys_open+0x73/0xa0 [ 4832.056108] __x64_sys_openat+0x24/0x30 [ 4832.056111] do_syscall_64+0xc1/0x440 [ 4832.056118] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4832.056124] RIP: 0033:0x7fdc0afbf332 [ 4832.056130] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4832.056134] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4832.056139] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4832.056141] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4832.056143] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4832.056146] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4832.056148] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4832.068197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4832.068286] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4832.068294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4832.068297] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4832.068312] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4832.068316] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4832.068320] RAX: 0000000000007f16 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4832.068322] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4832.068325] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4832.068327] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4832.068329] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4832.068332] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4832.068335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4832.068337] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4832.068345] Call Trace: [ 4832.068351] [ 4832.068357] ? show_regs.cold.9+0x22/0x2f [ 4832.068367] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4832.068374] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4832.068378] ? watchdog+0x40/0x40 [ 4832.068382] ? hrtimer_interrupt+0x128/0x2c0 [ 4832.068386] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4832.068394] ? apic_timer_interrupt+0xf/0x20 [ 4832.068398] [ 4832.068401] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4832.068405] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4832.068408] ? dump_stack+0xf6/0x10e [ 4832.068413] do_raw_spin_lock+0x177/0x190 [ 4832.068417] _raw_spin_lock+0x21/0x30 [ 4832.068422] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4832.068688] obd_mod_free+0x46/0x140 [ptlrpc] [ 4832.068885] mdc_commit_open+0x91/0xb0 [mdc] [ 4832.068922] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4832.069285] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4832.069497] after_reply+0xe2c/0x1780 [ptlrpc] [ 4832.069713] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4832.076631] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4832.080067] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4832.080077] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4832.080095] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4832.080396] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4832.080407] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4832.080412] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4832.080416] RAX: 0000000000006982 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4832.080418] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4832.080420] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4832.080422] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4832.080424] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4832.080427] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4832.080430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4832.080432] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4832.080441] Call Trace: [ 4832.080446] [ 4832.080452] ? show_regs.cold.9+0x22/0x2f [ 4832.080463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4832.080469] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4832.080475] ? watchdog+0x40/0x40 [ 4832.080479] ? hrtimer_interrupt+0x128/0x2c0 [ 4832.080483] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4832.080492] ? apic_timer_interrupt+0xf/0x20 [ 4832.080496] [ 4832.080499] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4832.080504] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4832.080510] do_raw_spin_lock+0x177/0x190 [ 4832.080514] _raw_spin_lock+0x21/0x30 [ 4832.080520] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4832.080730] process_one_work+0x2c8/0x700 [ 4832.080738] worker_thread+0x296/0x6e0 [ 4832.080743] ? rescuer_thread+0x570/0x570 [ 4832.080746] kthread+0x1d1/0x200 [ 4832.080751] ? set_kthread_struct+0x70/0x70 [ 4832.080755] ret_from_fork+0x1f/0x30 [ 4832.967365] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4832.970601] ? do_wait_intr+0xf0/0xf0 [ 4832.973082] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4832.976308] kthread+0x1d1/0x200 [ 4832.978527] ? set_kthread_struct+0x70/0x70 [ 4832.981036] ret_from_fork+0x1f/0x30 [ 4835.299482] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2810s! [ 4835.315529] Showing busy workqueues and worker pools: [ 4835.319787] workqueue events: flags=0x0 [ 4835.327767] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4835.343558] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4835.358924] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4835.366073] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4835.381503] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4835.394136] pending: kfree_rcu_monitor [ 4835.400736] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4835.411023] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4835.429312] workqueue events_long: flags=0x0 [ 4835.441015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4835.458277] pending: lu_objects_flush [obdclass] [ 4835.465012] workqueue events_unbound: flags=0x2 [ 4835.471028] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4835.478113] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4835.492552] workqueue events_power_efficient: flags=0x80 [ 4835.502060] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4835.505293] pending: neigh_periodic_work, check_lifetime [ 4835.511233] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4835.520672] pending: do_cache_clean, neigh_periodic_work [ 4835.522454] workqueue mm_percpu_wq: flags=0x8 [ 4835.530146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4835.543677] pending: lru_add_drain_per_cpu [ 4835.552100] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4835.562908] pending: vmstat_update, lru_add_drain_per_cpu [ 4835.571650] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4835.578118] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4835.585292] workqueue writeback: flags=0x4a [ 4835.591649] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4835.606417] pending: wb_workfn, wb_workfn [ 4835.607955] workqueue xprtiod: flags=0xa [ 4835.611746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4835.614081] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4835.620620] workqueue ipv6_addrconf: flags=0x40008 [ 4835.624603] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4835.629243] pending: addrconf_verify_work [ 4835.632953] workqueue knbd0-recv: flags=0x1a [ 4835.635857] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4835.641178] in-flight: 655:recv_work [ 4835.645026] workqueue ptlrpc_pinger: flags=0x2 [ 4835.650093] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4835.655748] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4835.660535] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2810s workers=4 idle: 36396 38 11188 [ 4835.663709] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4826s workers=2 idle: 658 [ 4848.054016] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4848.064033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4848.116129] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4848.126039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4848.136647] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4848.141646] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4848.162520] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4848.170422] RAX: 0000000000007988 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4848.179902] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4848.188236] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4848.195228] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4848.206993] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4848.213839] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4848.218033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4848.224509] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4848.233572] Call Trace: [ 4848.234370] [ 4848.235093] ? show_regs.cold.9+0x22/0x2f [ 4848.236348] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4848.243251] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4848.250606] ? watchdog+0x40/0x40 [ 4848.251903] ? hrtimer_interrupt+0x128/0x2c0 [ 4848.253297] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4848.263616] ? apic_timer_interrupt+0xf/0x20 [ 4848.266456] [ 4848.267533] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4848.270997] do_raw_spin_lock+0x177/0x190 [ 4848.275693] _raw_spin_lock+0x21/0x30 [ 4848.280511] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4848.286439] ? do_raw_spin_unlock+0x75/0x190 [ 4848.289477] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4848.294599] ? do_wait_intr+0xf0/0xf0 [ 4848.297486] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4848.302370] mdc_close+0x53b/0xd70 [mdc] [ 4848.306724] lmv_close+0x18d/0x490 [lmv] [ 4848.311601] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4848.318170] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4848.325778] ll_md_close+0x274/0x800 [lustre] [ 4848.332883] ll_file_release+0x128/0xc50 [lustre] [ 4848.340770] __fput+0x10d/0x440 [ 4848.343018] ____fput+0x12/0x20 [ 4848.345222] task_work_run+0xc6/0x110 [ 4848.347411] exit_to_usermode_loop+0x1dd/0x1f0 [ 4848.350978] do_syscall_64+0x3d6/0x440 [ 4848.353595] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4848.361200] RIP: 0033:0x7fdc0afbfcd8 [ 4848.363773] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4848.390102] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4848.392812] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4848.404159] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4848.412136] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4848.421768] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4848.432705] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4860.049687] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 4860.053404] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 4860.054766] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4860.054843] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4860.054849] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4860.054852] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4860.054867] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4860.054870] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4860.054875] RAX: 000000000000431a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4860.054877] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4860.054880] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4860.054882] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4860.054884] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4860.054887] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4860.054890] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4860.054893] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4860.054901] Call Trace: [ 4860.054906] [ 4860.054912] ? show_regs.cold.9+0x22/0x2f [ 4860.054922] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4860.054928] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4860.054932] ? watchdog+0x40/0x40 [ 4860.054937] ? hrtimer_interrupt+0x128/0x2c0 [ 4860.054941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4860.054950] ? apic_timer_interrupt+0xf/0x20 [ 4860.054954] [ 4860.054958] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4860.054962] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4860.054966] ? dump_stack+0xf6/0x10e [ 4860.054972] do_raw_spin_lock+0x177/0x190 [ 4860.054977] _raw_spin_lock+0x21/0x30 [ 4860.054984] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4860.055667] obd_mod_free+0x46/0x140 [ptlrpc] [ 4860.055884] mdc_commit_open+0x91/0xb0 [mdc] [ 4860.055975] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 4860.055980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4860.056041] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4860.056045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4860.056047] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4860.056055] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4860.056059] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4860.056063] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4860.056066] RDX: 0000000000004563 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4860.056068] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4860.056083] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4860.056085] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4860.056088] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4860.056091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4860.056094] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4860.056192] Call Trace: [ 4860.056196] [ 4860.056198] ? show_regs.cold.9+0x22/0x2f [ 4860.056207] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4860.056212] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4860.056216] ? watchdog+0x40/0x40 [ 4860.056220] ? hrtimer_interrupt+0x128/0x2c0 [ 4860.056224] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4860.056231] ? apic_timer_interrupt+0xf/0x20 [ 4860.056235] [ 4860.056238] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4860.056244] do_raw_spin_lock+0x177/0x190 [ 4860.056248] _raw_spin_lock+0x21/0x30 [ 4860.056253] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4860.056462] ? do_raw_spin_unlock+0x75/0x190 [ 4860.056468] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4860.056662] ? do_wait_intr+0xf0/0xf0 [ 4860.056671] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4860.056908] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4860.057599] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4860.057854] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.057987] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4860.058023] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4860.058052] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4860.058086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.058230] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4860.058464] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4860.058491] lmv_intent_open+0x338/0x1150 [lmv] [ 4860.058523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.058606] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.058688] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4860.058712] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4860.058825] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.058922] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4860.059017] ? do_raw_spin_unlock+0x75/0x190 [ 4860.059024] ? _raw_spin_unlock+0x12/0x30 [ 4860.059029] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4860.059060] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4860.059083] ll_atomic_open+0x36a/0xd70 [lustre] [ 4860.059315] lookup_open+0x356/0xb40 [ 4860.059324] do_last+0x313/0xfc0 [ 4860.059329] ? nd_jump_root+0xe5/0x160 [ 4860.059334] ? path_init+0x437/0x520 [ 4860.059338] path_openat+0xf7/0x500 [ 4860.059344] ? do_raw_spin_unlock+0x75/0x190 [ 4860.059348] do_filp_open+0x99/0x140 [ 4860.059355] ? getname_flags+0x6e/0x330 [ 4860.059359] ? __check_object_size+0xff/0x256 [ 4860.059366] ? do_raw_spin_unlock+0x75/0x190 [ 4860.059370] ? _raw_spin_unlock+0x12/0x30 [ 4860.059375] do_sys_openat2+0x2b4/0x410 [ 4860.059379] do_sys_open+0x73/0xa0 [ 4860.059383] __x64_sys_openat+0x24/0x30 [ 4860.059387] do_syscall_64+0xc1/0x440 [ 4860.059393] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4860.059399] RIP: 0033:0x7fdc0afbf332 [ 4860.059406] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4860.059410] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4860.059415] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4860.059418] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4860.059421] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4860.059423] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4860.059425] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4860.065750] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4860.065949] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4860.065957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4860.065963] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4860.066609] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4860.066621] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4860.066625] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4860.066630] RAX: 0000000000005d87 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4860.066632] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4860.066634] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4860.066636] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4860.066639] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4860.066642] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4860.066645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4860.066647] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4860.066656] Call Trace: [ 4860.066661] [ 4860.066666] ? show_regs.cold.9+0x22/0x2f [ 4860.066676] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4860.066682] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4860.066686] ? watchdog+0x40/0x40 [ 4860.066691] ? hrtimer_interrupt+0x128/0x2c0 [ 4860.066695] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4860.066703] ? apic_timer_interrupt+0xf/0x20 [ 4860.066708] [ 4860.066711] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4860.066716] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4860.066722] do_raw_spin_lock+0x177/0x190 [ 4860.066727] _raw_spin_lock+0x21/0x30 [ 4860.066732] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4860.067014] process_one_work+0x2c8/0x700 [ 4860.067159] worker_thread+0x296/0x6e0 [ 4860.067165] ? rescuer_thread+0x570/0x570 [ 4860.067168] kthread+0x1d1/0x200 [ 4860.067173] ? set_kthread_struct+0x70/0x70 [ 4860.067177] ret_from_fork+0x1f/0x30 [ 4860.929721] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4860.934028] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4860.939279] after_reply+0xe2c/0x1780 [ptlrpc] [ 4860.942600] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4860.947290] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4860.951527] ? do_wait_intr+0xf0/0xf0 [ 4860.956712] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4860.960718] kthread+0x1d1/0x200 [ 4860.961774] ? set_kthread_struct+0x70/0x70 [ 4860.963676] ret_from_fork+0x1f/0x30 [ 4866.015061] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2841s! [ 4866.017836] Showing busy workqueues and worker pools: [ 4866.026355] workqueue events: flags=0x0 [ 4866.028627] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4866.032807] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4866.041033] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4866.050825] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4866.054857] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4866.066931] pending: kfree_rcu_monitor [ 4866.070342] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4866.072462] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4866.096899] workqueue events_long: flags=0x0 [ 4866.101504] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4866.109624] pending: lu_objects_flush [obdclass] [ 4866.116698] workqueue events_unbound: flags=0x2 [ 4866.121067] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4866.129534] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4866.141732] workqueue events_power_efficient: flags=0x80 [ 4866.150448] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4866.162914] pending: neigh_periodic_work, check_lifetime [ 4866.171467] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4866.181082] pending: do_cache_clean, neigh_periodic_work [ 4866.183209] workqueue mm_percpu_wq: flags=0x8 [ 4866.201857] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4866.214909] pending: lru_add_drain_per_cpu [ 4866.219149] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4866.229453] pending: vmstat_update, lru_add_drain_per_cpu [ 4866.234586] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4866.247190] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4866.263947] workqueue writeback: flags=0x4a [ 4866.273159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4866.281012] pending: wb_workfn, wb_workfn [ 4866.283086] workqueue xprtiod: flags=0xa [ 4866.292356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4866.304843] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4866.320022] workqueue ipv6_addrconf: flags=0x40008 [ 4866.329702] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4866.343688] pending: addrconf_verify_work [ 4866.354214] workqueue knbd0-recv: flags=0x1a [ 4866.365032] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4866.377288] in-flight: 655:recv_work [ 4866.390856] workqueue ptlrpc_pinger: flags=0x2 [ 4866.394010] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4866.401997] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4866.411812] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2841s workers=4 idle: 36396 38 11188 [ 4866.423717] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4856s workers=2 idle: 658 [ 4876.054027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4876.057410] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4876.075018] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4876.081206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4876.087783] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4876.089172] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4876.095149] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4876.097918] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4876.100419] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4876.102516] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4876.104693] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4876.107142] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4876.109415] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4876.112369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4876.114272] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4876.116834] Call Trace: [ 4876.117752] [ 4876.118597] ? show_regs.cold.9+0x22/0x2f [ 4876.119966] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4876.121787] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4876.123308] ? watchdog+0x40/0x40 [ 4876.124502] ? hrtimer_interrupt+0x128/0x2c0 [ 4876.125975] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4876.128371] ? apic_timer_interrupt+0xf/0x20 [ 4876.129786] [ 4876.130569] ? native_safe_halt+0x2e/0x50 [ 4876.131921] kvm_wait+0xcd/0xe0 [ 4876.133240] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4876.135276] do_raw_spin_lock+0x177/0x190 [ 4876.136838] _raw_spin_lock+0x21/0x30 [ 4876.138294] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4876.140464] ? do_raw_spin_unlock+0x75/0x190 [ 4876.142696] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4876.145022] ? do_wait_intr+0xf0/0xf0 [ 4876.146530] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4876.148916] mdc_close+0x53b/0xd70 [mdc] [ 4876.150189] lmv_close+0x18d/0x490 [lmv] [ 4876.151905] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4876.154493] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4876.156695] ll_md_close+0x274/0x800 [lustre] [ 4876.158469] ll_file_release+0x128/0xc50 [lustre] [ 4876.160509] __fput+0x10d/0x440 [ 4876.161658] ____fput+0x12/0x20 [ 4876.162781] task_work_run+0xc6/0x110 [ 4876.164356] exit_to_usermode_loop+0x1dd/0x1f0 [ 4876.165937] do_syscall_64+0x3d6/0x440 [ 4876.167176] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4876.169436] RIP: 0033:0x7fdc0afbfcd8 [ 4876.170720] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4876.186995] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4876.190334] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4876.193518] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4876.202956] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4876.209263] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4876.211962] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4888.051827] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4888.052024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4888.055227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [ 4888.058624] Modules linked in: [ 4888.058626] ptlrpc_gss(O) ptlrpc(O) [ 4888.064706] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4888.064789] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4888.064797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4888.064799] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4888.064813] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4888.064818] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4888.064823] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4888.064825] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4888.064827] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4888.064830] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4888.064832] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4888.064835] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4888.064838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4888.064841] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4888.064849] Call Trace: [ 4888.064854] [ 4888.064859] ? show_regs.cold.9+0x22/0x2f [ 4888.064870] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4888.064878] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4888.064882] ? watchdog+0x40/0x40 [ 4888.064886] ? hrtimer_interrupt+0x128/0x2c0 [ 4888.064897] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4888.064907] ? apic_timer_interrupt+0xf/0x20 [ 4888.064911] [ 4888.064913] ? native_safe_halt+0x2e/0x50 [ 4888.064916] kvm_wait+0xcd/0xe0 [ 4888.064923] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4888.064930] do_raw_spin_lock+0x177/0x190 [ 4888.064934] _raw_spin_lock+0x21/0x30 [ 4888.064939] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4888.069256] obdclass(O) [ 4888.070621] ? do_raw_spin_unlock+0x75/0x190 [ 4888.107530] ksocklnd(O) [ 4888.112609] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4888.112935] ? do_wait_intr+0xf0/0xf0 [ 4888.112946] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4888.113277] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4888.113513] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4888.122804] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4888.122854] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4888.122921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4888.122928] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4888.123222] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4888.123231] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4888.123235] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4888.123240] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4888.123242] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4888.123245] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4888.123247] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4888.123250] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4888.123253] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4888.123256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4888.123258] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4888.123273] Call Trace: [ 4888.123278] [ 4888.123283] ? show_regs.cold.9+0x22/0x2f [ 4888.123294] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4888.123300] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4888.123306] ? watchdog+0x40/0x40 [ 4888.123310] ? hrtimer_interrupt+0x128/0x2c0 [ 4888.123314] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4888.123321] ? apic_timer_interrupt+0xf/0x20 [ 4888.123326] [ 4888.123328] ? native_safe_halt+0x2e/0x50 [ 4888.123333] kvm_wait+0xcd/0xe0 [ 4888.123342] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4888.123350] do_raw_spin_lock+0x177/0x190 [ 4888.123355] _raw_spin_lock+0x21/0x30 [ 4888.123359] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4888.123996] process_one_work+0x2c8/0x700 [ 4888.124004] worker_thread+0x296/0x6e0 [ 4888.124009] ? rescuer_thread+0x570/0x570 [ 4888.124013] kthread+0x1d1/0x200 [ 4888.124018] ? set_kthread_struct+0x70/0x70 [ 4888.124022] ret_from_fork+0x1f/0x30 [ 4888.284446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4888.286454] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4888.288514] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4888.290418] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4888.292154] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4888.294397] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4888.296994] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4888.299713] lmv_intent_open+0x338/0x1150 [lmv] [ 4888.301709] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4888.303657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4888.305993] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4888.307944] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4888.309977] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4888.312318] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4888.314363] ? do_raw_spin_unlock+0x75/0x190 [ 4888.317147] ? _raw_spin_unlock+0x12/0x30 [ 4888.318779] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4888.322409] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4888.326556] ll_atomic_open+0x36a/0xd70 [lustre] [ 4888.334070] lookup_open+0x356/0xb40 [ 4888.335664] do_last+0x313/0xfc0 [ 4888.337219] ? nd_jump_root+0xe5/0x160 [ 4888.339076] ? path_init+0x437/0x520 [ 4888.340441] path_openat+0xf7/0x500 [ 4888.341907] ? do_raw_spin_unlock+0x75/0x190 [ 4888.343781] do_filp_open+0x99/0x140 [ 4888.345394] ? getname_flags+0x6e/0x330 [ 4888.347124] ? __check_object_size+0xff/0x256 [ 4888.349959] ? do_raw_spin_unlock+0x75/0x190 [ 4888.352632] ? _raw_spin_unlock+0x12/0x30 [ 4888.357122] do_sys_openat2+0x2b4/0x410 [ 4888.361457] do_sys_open+0x73/0xa0 [ 4888.366908] __x64_sys_openat+0x24/0x30 [ 4888.374402] do_syscall_64+0xc1/0x440 [ 4888.383387] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4888.391673] RIP: 0033:0x7fdc0afbf332 [ 4888.395120] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4888.409826] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4888.420606] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4888.424245] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4888.433308] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4888.437099] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4888.443017] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4892.048030] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 4892.052367] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4892.095828] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4892.111126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4892.117355] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4892.126398] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4892.145643] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4892.152532] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4892.159670] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4892.165545] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4892.173102] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4892.181722] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4892.185973] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4892.193011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4892.196239] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4892.199430] Call Trace: [ 4892.200306] [ 4892.201033] ? show_regs.cold.9+0x22/0x2f [ 4892.203362] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4892.208316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4892.218647] ? watchdog+0x40/0x40 [ 4892.226007] ? hrtimer_interrupt+0x128/0x2c0 [ 4892.228685] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4892.234866] ? apic_timer_interrupt+0xf/0x20 [ 4892.238470] [ 4892.239289] ? native_safe_halt+0x2e/0x50 [ 4892.241340] kvm_wait+0xcd/0xe0 [ 4892.242525] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4892.244864] ? dump_stack+0xf6/0x10e [ 4892.248070] do_raw_spin_lock+0x177/0x190 [ 4892.251352] _raw_spin_lock+0x21/0x30 [ 4892.254028] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4892.256992] obd_mod_free+0x46/0x140 [ptlrpc] [ 4892.259348] mdc_commit_open+0x91/0xb0 [mdc] [ 4892.261241] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4892.263185] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4892.265630] after_reply+0xe2c/0x1780 [ptlrpc] [ 4892.268207] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4892.270966] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4892.273143] ? do_wait_intr+0xf0/0xf0 [ 4892.278370] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4892.283993] kthread+0x1d1/0x200 [ 4892.285459] ? set_kthread_struct+0x70/0x70 [ 4892.288028] ret_from_fork+0x1f/0x30 [ 4896.735047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2871s! [ 4896.739205] Showing busy workqueues and worker pools: [ 4896.742452] workqueue events: flags=0x0 [ 4896.745016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4896.748407] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4896.752694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4896.756324] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4896.761928] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4896.765174] pending: kfree_rcu_monitor [ 4896.768273] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4896.772029] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4896.776847] workqueue events_long: flags=0x0 [ 4896.778721] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4896.781210] pending: lu_objects_flush [obdclass] [ 4896.783339] workqueue events_unbound: flags=0x2 [ 4896.785316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4896.788110] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4896.792336] workqueue events_power_efficient: flags=0x80 [ 4896.794227] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4896.796494] pending: neigh_periodic_work, check_lifetime [ 4896.798760] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4896.801559] pending: do_cache_clean, neigh_periodic_work [ 4896.803785] workqueue mm_percpu_wq: flags=0x8 [ 4896.805573] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4896.808421] pending: lru_add_drain_per_cpu [ 4896.810419] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4896.814014] pending: vmstat_update, lru_add_drain_per_cpu [ 4896.817136] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4896.821145] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4896.826053] workqueue writeback: flags=0x4a [ 4896.829272] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4896.831678] pending: wb_workfn, wb_workfn [ 4896.833482] workqueue xprtiod: flags=0xa [ 4896.835034] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4896.837532] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4896.840029] workqueue ipv6_addrconf: flags=0x40008 [ 4896.842078] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4896.847869] pending: addrconf_verify_work [ 4896.849500] workqueue knbd0-recv: flags=0x1a [ 4896.853659] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4896.861310] in-flight: 655:recv_work [ 4896.862634] workqueue ptlrpc_pinger: flags=0x2 [ 4896.864374] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4896.867366] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4896.869955] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2871s workers=4 idle: 36396 38 11188 [ 4896.877012] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4887s workers=2 idle: 658 [ 4904.054020] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4904.058227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4904.082371] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4904.088823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4904.093507] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4904.097062] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4904.105685] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4904.109123] RAX: 0000000000004b54 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4904.112111] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4904.115658] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4904.119220] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4904.122112] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4904.125972] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4904.129832] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4904.132878] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4904.136444] Call Trace: [ 4904.137516] [ 4904.138815] ? show_regs.cold.9+0x22/0x2f [ 4904.140592] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4904.142458] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4904.144903] ? watchdog+0x40/0x40 [ 4904.146748] ? hrtimer_interrupt+0x128/0x2c0 [ 4904.148829] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4904.151387] ? apic_timer_interrupt+0xf/0x20 [ 4904.153419] [ 4904.154389] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4904.157131] do_raw_spin_lock+0x177/0x190 [ 4904.158699] _raw_spin_lock+0x21/0x30 [ 4904.160527] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4904.163507] ? do_raw_spin_unlock+0x75/0x190 [ 4904.165498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4904.168576] ? do_wait_intr+0xf0/0xf0 [ 4904.170649] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4904.173347] mdc_close+0x53b/0xd70 [mdc] [ 4904.175165] lmv_close+0x18d/0x490 [lmv] [ 4904.177471] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4904.179783] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4904.181572] ll_md_close+0x274/0x800 [lustre] [ 4904.183704] ll_file_release+0x128/0xc50 [lustre] [ 4904.186564] __fput+0x10d/0x440 [ 4904.188240] ____fput+0x12/0x20 [ 4904.191789] task_work_run+0xc6/0x110 [ 4904.195465] exit_to_usermode_loop+0x1dd/0x1f0 [ 4904.198024] do_syscall_64+0x3d6/0x440 [ 4904.199817] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4904.201792] RIP: 0033:0x7fdc0afbfcd8 [ 4904.203459] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4904.210770] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4904.213741] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4904.216962] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4904.220860] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4904.223928] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4904.226871] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4916.052164] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4916.054463] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4916.055688] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4916.055782] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4916.055788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4916.055791] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4916.055804] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4916.055808] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4916.055813] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4916.055815] RDX: 00000000000056f4 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4916.055818] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4916.055821] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4916.055823] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4916.055827] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4916.055830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4916.055832] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4916.055841] Call Trace: [ 4916.055846] [ 4916.055851] ? show_regs.cold.9+0x22/0x2f [ 4916.055860] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4916.055867] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4916.055873] ? watchdog+0x40/0x40 [ 4916.055877] ? hrtimer_interrupt+0x128/0x2c0 [ 4916.055882] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4916.055892] ? apic_timer_interrupt+0xf/0x20 [ 4916.055896] [ 4916.055899] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4916.055904] do_raw_spin_lock+0x177/0x190 [ 4916.055910] _raw_spin_lock+0x21/0x30 [ 4916.055916] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4916.056517] ? do_raw_spin_unlock+0x75/0x190 [ 4916.056523] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4916.056671] ? do_wait_intr+0xf0/0xf0 [ 4916.056678] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4916.056955] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4916.067132] Modules linked in: [ 4916.088625] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4916.089481] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4916.089570] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4916.089649] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4916.089679] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4916.089713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4916.089805] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4916.089997] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4916.106682] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 4916.110227] lmv_intent_open+0x338/0x1150 [lmv] [ 4916.110264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4916.110353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4916.110409] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4916.110423] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4916.110479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4916.110534] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4916.110588] ? do_raw_spin_unlock+0x75/0x190 [ 4916.110594] ? _raw_spin_unlock+0x12/0x30 [ 4916.110599] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4916.110619] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4916.110633] ll_atomic_open+0x36a/0xd70 [lustre] [ 4916.110707] lookup_open+0x356/0xb40 [ 4916.110716] do_last+0x313/0xfc0 [ 4916.110719] ? nd_jump_root+0xe5/0x160 [ 4916.110723] ? path_init+0x437/0x520 [ 4916.110727] path_openat+0xf7/0x500 [ 4916.110732] ? do_raw_spin_unlock+0x75/0x190 [ 4916.110736] do_filp_open+0x99/0x140 [ 4916.110743] ? getname_flags+0x6e/0x330 [ 4916.110747] ? __check_object_size+0xff/0x256 [ 4916.110755] ? do_raw_spin_unlock+0x75/0x190 [ 4916.110759] ? _raw_spin_unlock+0x12/0x30 [ 4916.110764] do_sys_openat2+0x2b4/0x410 [ 4916.110769] do_sys_open+0x73/0xa0 [ 4916.110783] __x64_sys_openat+0x24/0x30 [ 4916.110788] do_syscall_64+0xc1/0x440 [ 4916.110794] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4916.110800] RIP: 0033:0x7fdc0afbf332 [ 4916.110807] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4916.110812] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4916.110818] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4916.110821] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4916.110824] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4916.110826] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4916.110829] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4916.420927] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4916.445645] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4916.451296] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4916.455247] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4916.458104] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4916.460609] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4916.468845] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4916.472301] RAX: 0000000000006505 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4916.475230] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4916.478098] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4916.483221] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4916.492753] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4916.501834] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4916.507564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4916.516542] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4916.518694] Call Trace: [ 4916.521295] [ 4916.522740] ? show_regs.cold.9+0x22/0x2f [ 4916.524753] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4916.527201] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4916.532125] ? watchdog+0x40/0x40 [ 4916.535020] ? hrtimer_interrupt+0x128/0x2c0 [ 4916.537582] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4916.540888] ? apic_timer_interrupt+0xf/0x20 [ 4916.543821] [ 4916.545044] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4916.548227] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4916.550841] do_raw_spin_lock+0x177/0x190 [ 4916.553031] _raw_spin_lock+0x21/0x30 [ 4916.554758] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4916.557912] process_one_work+0x2c8/0x700 [ 4916.559716] worker_thread+0x296/0x6e0 [ 4916.561337] ? rescuer_thread+0x570/0x570 [ 4916.563066] kthread+0x1d1/0x200 [ 4916.564571] ? set_kthread_struct+0x70/0x70 [ 4916.566452] ret_from_fork+0x1f/0x30 [ 4924.051946] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4924.062542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4924.112466] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4924.121421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4924.132724] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4924.141085] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4924.159368] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4924.167330] RAX: 0000000000007d13 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4924.174606] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4924.178009] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4924.183304] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4924.187587] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4924.190831] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4924.195399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4924.199831] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4924.208145] Call Trace: [ 4924.211052] [ 4924.214889] ? show_regs.cold.9+0x22/0x2f [ 4924.216542] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4924.219433] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4924.223710] ? watchdog+0x40/0x40 [ 4924.229544] ? hrtimer_interrupt+0x128/0x2c0 [ 4924.231088] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4924.238420] ? apic_timer_interrupt+0xf/0x20 [ 4924.246182] [ 4924.252118] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4924.253902] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4924.264560] ? dump_stack+0xf6/0x10e [ 4924.269288] do_raw_spin_lock+0x177/0x190 [ 4924.274070] _raw_spin_lock+0x21/0x30 [ 4924.277757] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4924.282004] obd_mod_free+0x46/0x140 [ptlrpc] [ 4924.286388] mdc_commit_open+0x91/0xb0 [mdc] [ 4924.288691] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4924.291776] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4924.296541] after_reply+0xe2c/0x1780 [ptlrpc] [ 4924.302685] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4924.305453] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4924.310164] ? do_wait_intr+0xf0/0xf0 [ 4924.316760] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4924.321329] kthread+0x1d1/0x200 [ 4924.324422] ? set_kthread_struct+0x70/0x70 [ 4924.331114] ret_from_fork+0x1f/0x30 [ 4926.946620] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2902s! [ 4926.958817] Showing busy workqueues and worker pools: [ 4926.965227] workqueue events: flags=0x0 [ 4926.966573] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4926.981793] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4926.990193] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4926.992548] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4926.996473] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4926.998658] pending: kfree_rcu_monitor [ 4927.000477] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4927.004049] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4927.009978] workqueue events_long: flags=0x0 [ 4927.011767] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4927.014862] pending: lu_objects_flush [obdclass] [ 4927.017412] workqueue events_unbound: flags=0x2 [ 4927.019565] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4927.022287] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4927.027639] workqueue events_power_efficient: flags=0x80 [ 4927.029957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4927.032235] pending: neigh_periodic_work, check_lifetime [ 4927.034562] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4927.037473] pending: do_cache_clean, neigh_periodic_work [ 4927.040008] workqueue mm_percpu_wq: flags=0x8 [ 4927.042126] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4927.045173] pending: lru_add_drain_per_cpu [ 4927.047325] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4927.050272] pending: vmstat_update, lru_add_drain_per_cpu [ 4927.052609] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4927.055696] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4927.059113] workqueue writeback: flags=0x4a [ 4927.060929] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4927.062953] pending: wb_workfn, wb_workfn [ 4927.064455] workqueue xprtiod: flags=0xa [ 4927.066217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4927.068806] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4927.072395] workqueue ipv6_addrconf: flags=0x40008 [ 4927.074474] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4927.077465] pending: addrconf_verify_work [ 4927.079730] workqueue knbd0-recv: flags=0x1a [ 4927.081061] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4927.083706] in-flight: 655:recv_work [ 4927.085670] workqueue ptlrpc_pinger: flags=0x2 [ 4927.088063] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4927.090950] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4927.093643] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2902s workers=4 idle: 36396 38 11188 [ 4927.097323] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4917s workers=2 idle: 658 [ 4932.055481] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 4932.062624] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4932.099646] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4932.107028] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4932.112656] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4932.115640] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4932.126682] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4932.133783] RAX: 0000000000007f14 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4932.138945] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4932.142386] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4932.145424] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4932.155610] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4932.158001] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4932.170808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4932.181035] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4932.188336] Call Trace: [ 4932.190448] [ 4932.193454] ? show_regs.cold.9+0x22/0x2f [ 4932.196501] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4932.201931] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4932.203540] ? watchdog+0x40/0x40 [ 4932.210992] ? hrtimer_interrupt+0x128/0x2c0 [ 4932.219319] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4932.223692] ? apic_timer_interrupt+0xf/0x20 [ 4932.227655] [ 4932.230460] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4932.236371] do_raw_spin_lock+0x177/0x190 [ 4932.241366] _raw_spin_lock+0x21/0x30 [ 4932.245950] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4932.251480] ? do_raw_spin_unlock+0x75/0x190 [ 4932.256319] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4932.263326] ? do_wait_intr+0xf0/0xf0 [ 4932.266459] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4932.271779] mdc_close+0x53b/0xd70 [mdc] [ 4932.276917] lmv_close+0x18d/0x490 [lmv] [ 4932.281273] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4932.284362] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4932.286313] ll_md_close+0x274/0x800 [lustre] [ 4932.288464] ll_file_release+0x128/0xc50 [lustre] [ 4932.290676] __fput+0x10d/0x440 [ 4932.292169] ____fput+0x12/0x20 [ 4932.293452] task_work_run+0xc6/0x110 [ 4932.294765] exit_to_usermode_loop+0x1dd/0x1f0 [ 4932.296704] do_syscall_64+0x3d6/0x440 [ 4932.301601] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4932.307270] RIP: 0033:0x7fdc0afbfcd8 [ 4932.308502] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4932.321406] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4932.324843] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4932.327787] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4932.331933] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4932.336157] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4932.339781] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4944.051027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4944.052454] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4944.053249] Modules linked in: [ 4944.055494] Modules linked in: [ 4944.055496] lustre(O) osp(O) ofd(O) [ 4944.058086] lustre(O) [ 4944.061741] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4944.061814] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4944.061822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4944.061829] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4944.063462] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4944.063531] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4944.063537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4944.063540] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4944.063552] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4944.063556] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4944.063561] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4944.063564] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4944.063566] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4944.063568] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4944.063570] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4944.063573] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4944.063576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4944.063578] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4944.063585] Call Trace: [ 4944.063590] [ 4944.063595] ? show_regs.cold.9+0x22/0x2f [ 4944.063605] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4944.063612] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4944.063617] ? watchdog+0x40/0x40 [ 4944.063622] ? hrtimer_interrupt+0x128/0x2c0 [ 4944.063626] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4944.063632] ? apic_timer_interrupt+0xf/0x20 [ 4944.063637] [ 4944.063640] ? native_safe_halt+0x2e/0x50 [ 4944.063643] kvm_wait+0xcd/0xe0 [ 4944.063649] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4944.063656] do_raw_spin_lock+0x177/0x190 [ 4944.063660] _raw_spin_lock+0x21/0x30 [ 4944.063665] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4944.064147] ? do_raw_spin_unlock+0x75/0x190 [ 4944.064154] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4944.064369] ? do_wait_intr+0xf0/0xf0 [ 4944.064378] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4944.064581] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4944.064875] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4944.065205] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4944.066111] [ 4944.066116] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4944.066127] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4944.066132] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4944.066136] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4944.066138] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4944.066140] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4944.066141] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4944.066144] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4944.066148] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4944.066150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4944.066153] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4944.066161] Call Trace: [ 4944.066164] [ 4944.066168] ? show_regs.cold.9+0x22/0x2f [ 4944.066177] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4944.066182] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4944.066186] ? watchdog+0x40/0x40 [ 4944.066189] ? hrtimer_interrupt+0x128/0x2c0 [ 4944.066193] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4944.066199] ? apic_timer_interrupt+0xf/0x20 [ 4944.066203] [ 4944.066206] ? native_safe_halt+0x2e/0x50 [ 4944.066211] kvm_wait+0xcd/0xe0 [ 4944.066218] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4944.066226] do_raw_spin_lock+0x177/0x190 [ 4944.066230] _raw_spin_lock+0x21/0x30 [ 4944.066235] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4944.066505] process_one_work+0x2c8/0x700 [ 4944.066512] worker_thread+0x296/0x6e0 [ 4944.066516] ? rescuer_thread+0x570/0x570 [ 4944.066519] kthread+0x1d1/0x200 [ 4944.066524] ? set_kthread_struct+0x70/0x70 [ 4944.066527] ret_from_fork+0x1f/0x30 [ 4944.319658] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4944.322404] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4944.326241] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4944.329235] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4944.332992] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4944.336608] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4944.339491] lmv_intent_open+0x338/0x1150 [lmv] [ 4944.342126] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4944.344826] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4944.348072] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4944.350292] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4944.353069] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4944.355833] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4944.358222] ? do_raw_spin_unlock+0x75/0x190 [ 4944.360397] ? _raw_spin_unlock+0x12/0x30 [ 4944.362580] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4944.365182] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4944.367575] ll_atomic_open+0x36a/0xd70 [lustre] [ 4944.370209] lookup_open+0x356/0xb40 [ 4944.371868] do_last+0x313/0xfc0 [ 4944.373450] ? nd_jump_root+0xe5/0x160 [ 4944.375169] ? path_init+0x437/0x520 [ 4944.376538] path_openat+0xf7/0x500 [ 4944.378296] ? do_raw_spin_unlock+0x75/0x190 [ 4944.380180] do_filp_open+0x99/0x140 [ 4944.382297] ? getname_flags+0x6e/0x330 [ 4944.384430] ? __check_object_size+0xff/0x256 [ 4944.386673] ? do_raw_spin_unlock+0x75/0x190 [ 4944.388888] ? _raw_spin_unlock+0x12/0x30 [ 4944.390957] do_sys_openat2+0x2b4/0x410 [ 4944.392945] do_sys_open+0x73/0xa0 [ 4944.394553] __x64_sys_openat+0x24/0x30 [ 4944.396475] do_syscall_64+0xc1/0x440 [ 4944.398565] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4944.401802] RIP: 0033:0x7fdc0afbf332 [ 4944.403728] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4944.413581] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4944.416438] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4944.418971] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4944.421510] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4944.423389] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4944.426265] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4952.048189] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4952.055366] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4952.097908] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4952.106874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4952.111462] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4952.115676] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 4952.124075] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4952.127359] RAX: 000000000000773f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4952.129743] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4952.131925] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4952.134909] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4952.138323] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4952.141436] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4952.145484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4952.148093] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4952.151153] Call Trace: [ 4952.155563] [ 4952.156347] ? show_regs.cold.9+0x22/0x2f [ 4952.157786] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4952.168506] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4952.170125] ? watchdog+0x40/0x40 [ 4952.174416] ? hrtimer_interrupt+0x128/0x2c0 [ 4952.178419] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4952.183680] ? apic_timer_interrupt+0xf/0x20 [ 4952.187528] [ 4952.189539] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4952.196450] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4952.205563] ? dump_stack+0xf6/0x10e [ 4952.207578] do_raw_spin_lock+0x177/0x190 [ 4952.209130] _raw_spin_lock+0x21/0x30 [ 4952.210426] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4952.214066] obd_mod_free+0x46/0x140 [ptlrpc] [ 4952.216954] mdc_commit_open+0x91/0xb0 [mdc] [ 4952.221513] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4952.230345] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4952.235544] after_reply+0xe2c/0x1780 [ptlrpc] [ 4952.238536] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4952.241847] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4952.245073] ? do_wait_intr+0xf0/0xf0 [ 4952.250423] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4952.253678] kthread+0x1d1/0x200 [ 4952.255550] ? set_kthread_struct+0x70/0x70 [ 4952.257704] ret_from_fork+0x1f/0x30 [ 4957.160611] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2932s! [ 4957.167022] Showing busy workqueues and worker pools: [ 4957.168537] workqueue events: flags=0x0 [ 4957.169702] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4957.182126] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4957.186090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4957.194518] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4957.201712] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4957.209708] pending: kfree_rcu_monitor [ 4957.218646] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4957.227464] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4957.239504] workqueue events_long: flags=0x0 [ 4957.246047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4957.254198] pending: lu_objects_flush [obdclass] [ 4957.263963] workqueue events_unbound: flags=0x2 [ 4957.266274] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4957.275392] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4957.284615] workqueue events_power_efficient: flags=0x80 [ 4957.287103] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4957.304414] pending: neigh_periodic_work, check_lifetime [ 4957.307625] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4957.312985] pending: do_cache_clean, neigh_periodic_work [ 4957.321261] workqueue mm_percpu_wq: flags=0x8 [ 4957.324338] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4957.329415] pending: lru_add_drain_per_cpu [ 4957.335924] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4957.341885] pending: vmstat_update, lru_add_drain_per_cpu [ 4957.346011] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4957.356484] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4957.364089] workqueue writeback: flags=0x4a [ 4957.368912] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4957.377023] pending: wb_workfn, wb_workfn [ 4957.381594] workqueue xprtiod: flags=0xa [ 4957.383945] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4957.389609] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4957.395390] workqueue ipv6_addrconf: flags=0x40008 [ 4957.397150] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4957.403700] pending: addrconf_verify_work [ 4957.411558] workqueue knbd0-recv: flags=0x1a [ 4957.413199] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4957.424733] in-flight: 655:recv_work [ 4957.434261] workqueue ptlrpc_pinger: flags=0x2 [ 4957.441173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4957.451629] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4957.458099] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2932s workers=4 idle: 36396 38 11188 [ 4957.466862] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4948s workers=2 idle: 658 [ 4959.409060] rcu: INFO: rcu_sched self-detected stall on CPU [ 4959.414845] rcu: 0-...!: (1753145 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 4959.423789] (t=2940066 jiffies g=270761 q=3423) [ 4959.429257] rcu: rcu_sched kthread starved for 2934998 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 4959.442455] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4959.449954] rcu: RCU grace-period kthread stack dump: [ 4959.458503] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4959.469481] Call Trace: [ 4959.473206] __schedule+0x351/0xcb0 [ 4959.476819] schedule+0xc0/0x180 [ 4959.478235] schedule_timeout+0xb4/0x190 [ 4959.479877] ? do_raw_spin_unlock+0x75/0x190 [ 4959.483502] ? __next_timer_interrupt+0x160/0x160 [ 4959.484842] ? prepare_to_swait_event+0xf3/0x140 [ 4959.492121] rcu_gp_kthread+0x8dd/0x15b0 [ 4959.494537] ? force_qs_rnp+0x2c0/0x2c0 [ 4959.495827] kthread+0x1d1/0x200 [ 4959.496883] ? set_kthread_struct+0x70/0x70 [ 4959.498555] ret_from_fork+0x1f/0x30 [ 4959.501259] rcu: Stack dump where RCU GP kthread last ran: [ 4959.503778] Sending NMI from CPU 0 to CPUs 1: [ 4959.506210] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 4959.508310] NMI backtrace for cpu 0 [ 4959.511589] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4959.517168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4959.521554] Call Trace: [ 4959.522541] [ 4959.523443] ? dump_stack+0xbb/0x10e [ 4959.524759] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4959.529065] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4959.530822] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4959.537286] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4959.543409] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4959.548496] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4959.554343] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4959.561694] ? update_process_times+0x6d/0xc0 [ 4959.565411] ? tick_sched_handle.isra.18+0x27/0x90 [ 4959.571466] ? tick_sched_timer+0x53/0xb0 [ 4959.574602] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4959.577818] ? tick_sched_handle.isra.18+0x90/0x90 [ 4959.581872] ? hrtimer_interrupt+0x128/0x2c0 [ 4959.584121] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4959.586697] ? apic_timer_interrupt+0xf/0x20 [ 4959.589467] [ 4959.590443] ? native_safe_halt+0x2e/0x50 [ 4959.592282] ? kvm_wait+0xcd/0xe0 [ 4959.594152] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4959.597099] ? dump_stack+0xf6/0x10e [ 4959.600162] ? do_raw_spin_lock+0x177/0x190 [ 4959.603204] ? _raw_spin_lock+0x21/0x30 [ 4959.605775] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4959.609978] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4959.615551] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4959.617832] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4959.620829] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4959.624318] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4959.627807] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4959.632733] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4959.635884] ? do_wait_intr+0xf0/0xf0 [ 4959.638504] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4959.642467] ? kthread+0x1d1/0x200 [ 4959.646441] ? set_kthread_struct+0x70/0x70 [ 4959.649115] ? ret_from_fork+0x1f/0x30 [ 4960.062685] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4960.068700] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4960.118266] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4960.131645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4960.141506] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4960.153061] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4960.181235] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4960.186807] RAX: 0000000000006c0b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4960.190185] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4960.197819] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4960.201856] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4960.208890] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4960.215843] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4960.223327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4960.230330] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4960.232924] Call Trace: [ 4960.235702] [ 4960.238264] ? show_regs.cold.9+0x22/0x2f [ 4960.242692] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4960.244942] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4960.247934] ? watchdog+0x40/0x40 [ 4960.250036] ? hrtimer_interrupt+0x128/0x2c0 [ 4960.255628] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4960.262511] ? apic_timer_interrupt+0xf/0x20 [ 4960.264106] [ 4960.264859] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4960.274695] do_raw_spin_lock+0x177/0x190 [ 4960.278896] _raw_spin_lock+0x21/0x30 [ 4960.280923] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4960.287466] ? do_raw_spin_unlock+0x75/0x190 [ 4960.289770] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4960.295774] ? do_wait_intr+0xf0/0xf0 [ 4960.303035] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4960.309886] mdc_close+0x53b/0xd70 [mdc] [ 4960.317047] lmv_close+0x18d/0x490 [lmv] [ 4960.320767] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4960.324740] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4960.330615] ll_md_close+0x274/0x800 [lustre] [ 4960.339125] ll_file_release+0x128/0xc50 [lustre] [ 4960.341877] __fput+0x10d/0x440 [ 4960.344788] ____fput+0x12/0x20 [ 4960.348310] task_work_run+0xc6/0x110 [ 4960.350499] exit_to_usermode_loop+0x1dd/0x1f0 [ 4960.353315] do_syscall_64+0x3d6/0x440 [ 4960.358293] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4960.364534] RIP: 0033:0x7fdc0afbfcd8 [ 4960.367021] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4960.394868] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4960.402514] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4960.406944] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4960.416843] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4960.422749] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4960.428902] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4972.052062] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 4972.058788] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4972.111648] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4972.124565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4972.133846] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4972.143359] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4972.174545] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4972.187961] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 4972.196531] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 4972.204608] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 4972.211501] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 4972.223529] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4972.229047] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 4972.238774] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4972.243103] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 4972.248755] Call Trace: [ 4972.250870] [ 4972.251558] ? show_regs.cold.9+0x22/0x2f [ 4972.256657] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4972.258997] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4972.261841] ? watchdog+0x40/0x40 [ 4972.263726] ? hrtimer_interrupt+0x128/0x2c0 [ 4972.269222] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4972.273661] ? apic_timer_interrupt+0xf/0x20 [ 4972.276179] [ 4972.277297] ? native_safe_halt+0x2e/0x50 [ 4972.279916] kvm_wait+0xcd/0xe0 [ 4972.281189] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4972.283755] do_raw_spin_lock+0x177/0x190 [ 4972.286267] _raw_spin_lock+0x21/0x30 [ 4972.287918] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4972.291132] ? do_raw_spin_unlock+0x75/0x190 [ 4972.293698] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4972.296745] ? do_wait_intr+0xf0/0xf0 [ 4972.298410] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4972.301565] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4972.304296] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4972.307094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4972.309539] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4972.312151] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4972.314277] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4972.316365] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4972.318694] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4972.326554] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4972.329318] lmv_intent_open+0x338/0x1150 [lmv] [ 4972.331747] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4972.335589] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4972.337811] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4972.339942] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4972.341695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4972.344549] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4972.346727] ? do_raw_spin_unlock+0x75/0x190 [ 4972.348707] ? _raw_spin_unlock+0x12/0x30 [ 4972.350619] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4972.353159] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4972.355291] ll_atomic_open+0x36a/0xd70 [lustre] [ 4972.357497] lookup_open+0x356/0xb40 [ 4972.358962] do_last+0x313/0xfc0 [ 4972.360645] ? nd_jump_root+0xe5/0x160 [ 4972.362245] ? path_init+0x437/0x520 [ 4972.363969] path_openat+0xf7/0x500 [ 4972.365407] ? do_raw_spin_unlock+0x75/0x190 [ 4972.367481] do_filp_open+0x99/0x140 [ 4972.368974] ? getname_flags+0x6e/0x330 [ 4972.370909] ? __check_object_size+0xff/0x256 [ 4972.373192] ? do_raw_spin_unlock+0x75/0x190 [ 4972.375317] ? _raw_spin_unlock+0x12/0x30 [ 4972.377161] do_sys_openat2+0x2b4/0x410 [ 4972.379075] do_sys_open+0x73/0xa0 [ 4972.380638] __x64_sys_openat+0x24/0x30 [ 4972.382333] do_syscall_64+0xc1/0x440 [ 4972.384153] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4972.386359] RIP: 0033:0x7fdc0afbf332 [ 4972.387776] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4972.396093] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4972.400106] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 4972.405106] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 4972.409858] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 4972.417285] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 4972.427612] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 4984.048011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 4984.051559] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4984.056233] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 4984.073899] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4984.073912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4984.073915] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4984.073928] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4984.073933] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4984.073938] RAX: 0000000000004f9c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4984.073940] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 4984.073942] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 4984.073944] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 4984.073946] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 4984.073949] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 4984.073951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4984.073953] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 4984.073961] Call Trace: [ 4984.074106] [ 4984.074113] ? show_regs.cold.9+0x22/0x2f [ 4984.074124] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4984.074130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4984.074135] ? watchdog+0x40/0x40 [ 4984.074138] ? hrtimer_interrupt+0x128/0x2c0 [ 4984.074142] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4984.074150] ? apic_timer_interrupt+0xf/0x20 [ 4984.074155] [ 4984.074157] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4984.074173] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4984.074185] ? dump_stack+0xf6/0x10e [ 4984.074191] do_raw_spin_lock+0x177/0x190 [ 4984.074196] _raw_spin_lock+0x21/0x30 [ 4984.074204] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4984.074679] obd_mod_free+0x46/0x140 [ptlrpc] [ 4984.074865] mdc_commit_open+0x91/0xb0 [mdc] [ 4984.074905] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4984.075291] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4984.075661] after_reply+0xe2c/0x1780 [ptlrpc] [ 4984.075911] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4984.076121] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4984.076679] ? do_wait_intr+0xf0/0xf0 [ 4984.076690] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4984.076890] kthread+0x1d1/0x200 [ 4984.076896] ? set_kthread_struct+0x70/0x70 [ 4984.076899] ret_from_fork+0x1f/0x30 [ 4984.201015] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4984.223145] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4984.227237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4984.231014] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4984.233829] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4984.236624] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4984.246018] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4984.249664] RAX: 0000000000007066 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4984.253341] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 4984.256923] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4984.260412] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 4984.263867] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 4984.267382] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 4984.271444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4984.274206] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 4984.277676] Call Trace: [ 4984.279077] [ 4984.280133] ? show_regs.cold.9+0x22/0x2f [ 4984.281594] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4984.283521] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4984.285375] ? watchdog+0x40/0x40 [ 4984.286499] ? hrtimer_interrupt+0x128/0x2c0 [ 4984.288117] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4984.290448] ? apic_timer_interrupt+0xf/0x20 [ 4984.292882] [ 4984.293863] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4984.296662] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4984.299406] do_raw_spin_lock+0x177/0x190 [ 4984.301573] _raw_spin_lock+0x21/0x30 [ 4984.303515] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4984.306432] process_one_work+0x2c8/0x700 [ 4984.308424] worker_thread+0x296/0x6e0 [ 4984.310115] ? rescuer_thread+0x570/0x570 [ 4984.312138] kthread+0x1d1/0x200 [ 4984.313945] ? set_kthread_struct+0x70/0x70 [ 4984.315807] ret_from_fork+0x1f/0x30 [ 4987.875387] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2962s! [ 4987.880745] Showing busy workqueues and worker pools: [ 4987.891024] workqueue events: flags=0x0 [ 4987.895417] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4987.905349] pending: cpu_latency_work [ptlrpc], push_to_pool [ 4987.917396] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4987.928852] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 4987.941327] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4987.950874] pending: kfree_rcu_monitor [ 4987.952800] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4987.957970] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 4987.968264] workqueue events_long: flags=0x0 [ 4987.969913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4987.977561] pending: lu_objects_flush [obdclass] [ 4987.980064] workqueue events_unbound: flags=0x2 [ 4987.982074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4987.988837] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4987.994375] workqueue events_power_efficient: flags=0x80 [ 4987.996816] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4988.002700] pending: neigh_periodic_work, check_lifetime [ 4988.005575] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4988.010971] pending: do_cache_clean, neigh_periodic_work [ 4988.015391] workqueue mm_percpu_wq: flags=0x8 [ 4988.017409] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4988.023464] pending: lru_add_drain_per_cpu [ 4988.025777] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4988.031089] pending: vmstat_update, lru_add_drain_per_cpu [ 4988.043073] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4988.049639] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4988.054249] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 4988.057838] workqueue writeback: flags=0x4a [ 4988.060925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4988.061004] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4988.061010] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4988.061013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4988.061025] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4988.061031] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4988.061034] RAX: 000000000000788c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 4988.061036] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 4988.061037] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4988.061039] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 4988.061041] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 4988.061043] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 4988.061045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4988.061047] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 4988.061052] Call Trace: [ 4988.061057] [ 4988.061061] ? show_regs.cold.9+0x22/0x2f [ 4988.061069] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4988.061077] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4988.061080] ? watchdog+0x40/0x40 [ 4988.061082] ? hrtimer_interrupt+0x128/0x2c0 [ 4988.061085] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4988.061091] ? apic_timer_interrupt+0xf/0x20 [ 4988.061094] [ 4988.061096] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4988.061101] do_raw_spin_lock+0x177/0x190 [ 4988.061104] _raw_spin_lock+0x21/0x30 [ 4988.061107] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4988.061403] ? do_raw_spin_unlock+0x75/0x190 [ 4988.061408] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4988.061618] ? do_wait_intr+0xf0/0xf0 [ 4988.061625] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4988.061816] mdc_close+0x53b/0xd70 [mdc] [ 4988.061857] lmv_close+0x18d/0x490 [lmv] [ 4988.061886] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4988.062007] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4988.062076] ll_md_close+0x274/0x800 [lustre] [ 4988.062146] ll_file_release+0x128/0xc50 [lustre] [ 4988.062234] __fput+0x10d/0x440 [ 4988.062244] ____fput+0x12/0x20 [ 4988.062248] task_work_run+0xc6/0x110 [ 4988.062257] exit_to_usermode_loop+0x1dd/0x1f0 [ 4988.062264] do_syscall_64+0x3d6/0x440 [ 4988.062268] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4988.062276] RIP: 0033:0x7fdc0afbfcd8 [ 4988.062283] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4988.062288] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4988.062292] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 4988.062294] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 4988.062296] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 4988.062298] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 4988.062300] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4988.386260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4988.392732] pending: wb_workfn, wb_workfn [ 4988.398400] workqueue xprtiod: flags=0xa [ 4988.400860] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4988.404464] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4988.411544] workqueue ipv6_addrconf: flags=0x40008 [ 4988.415186] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4988.420255] pending: addrconf_verify_work [ 4988.426446] workqueue knbd0-recv: flags=0x1a [ 4988.434722] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4988.442945] in-flight: 655:recv_work [ 4988.445028] workqueue ptlrpc_pinger: flags=0x2 [ 4988.449711] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4988.454785] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 4988.458409] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2963s workers=4 idle: 36396 38 11188 [ 4988.464586] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4979s workers=2 idle: 658 [ 5000.053710] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5000.063642] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5000.115441] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5000.130333] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5000.147170] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5000.151264] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5000.163113] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5000.167340] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5000.170085] RDX: 0000000000003502 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5000.173870] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5000.177251] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5000.184273] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5000.196497] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5000.201772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5000.206097] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5000.211224] Call Trace: [ 5000.212431] [ 5000.213386] ? show_regs.cold.9+0x22/0x2f [ 5000.215596] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5000.217261] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5000.221872] ? watchdog+0x40/0x40 [ 5000.223475] ? hrtimer_interrupt+0x128/0x2c0 [ 5000.225992] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5000.228121] ? apic_timer_interrupt+0xf/0x20 [ 5000.230999] [ 5000.237491] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5000.246409] do_raw_spin_lock+0x177/0x190 [ 5000.249555] _raw_spin_lock+0x21/0x30 [ 5000.254580] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5000.258708] ? do_raw_spin_unlock+0x75/0x190 [ 5000.263793] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5000.266195] ? do_wait_intr+0xf0/0xf0 [ 5000.267105] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5000.269195] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5000.272270] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5000.275348] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5000.277938] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5000.284888] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5000.292749] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5000.305961] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5000.324279] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5000.336575] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5000.346524] lmv_intent_open+0x338/0x1150 [lmv] [ 5000.357324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5000.359153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5000.374434] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5000.381485] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5000.385136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5000.387610] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5000.390582] ? do_raw_spin_unlock+0x75/0x190 [ 5000.392161] ? _raw_spin_unlock+0x12/0x30 [ 5000.394031] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5000.396694] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5000.398280] ll_atomic_open+0x36a/0xd70 [lustre] [ 5000.400934] lookup_open+0x356/0xb40 [ 5000.403105] do_last+0x313/0xfc0 [ 5000.407456] ? nd_jump_root+0xe5/0x160 [ 5000.409357] ? path_init+0x437/0x520 [ 5000.411730] path_openat+0xf7/0x500 [ 5000.416162] ? do_raw_spin_unlock+0x75/0x190 [ 5000.418288] do_filp_open+0x99/0x140 [ 5000.420568] ? getname_flags+0x6e/0x330 [ 5000.424939] ? __check_object_size+0xff/0x256 [ 5000.435057] ? do_raw_spin_unlock+0x75/0x190 [ 5000.443451] ? _raw_spin_unlock+0x12/0x30 [ 5000.452737] do_sys_openat2+0x2b4/0x410 [ 5000.454040] do_sys_open+0x73/0xa0 [ 5000.459342] __x64_sys_openat+0x24/0x30 [ 5000.465113] do_syscall_64+0xc1/0x440 [ 5000.471582] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5000.477559] RIP: 0033:0x7fdc0afbf332 [ 5000.484092] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5000.512212] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5000.518364] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5000.529532] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5000.535331] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5000.538427] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5000.542120] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5012.051949] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5012.061086] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5012.105302] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5012.117307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5012.126046] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5012.132464] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5012.138564] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5012.155383] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5012.163245] RAX: 0000000000003114 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5012.171086] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5012.177805] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5012.190534] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5012.197105] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5012.206302] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5012.211178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5012.220396] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5012.224614] Call Trace: [ 5012.226078] [ 5012.227239] ? show_regs.cold.9+0x22/0x2f [ 5012.229428] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5012.232466] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5012.239107] ? watchdog+0x40/0x40 [ 5012.240573] ? hrtimer_interrupt+0x128/0x2c0 [ 5012.243356] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5012.253447] ? apic_timer_interrupt+0xf/0x20 [ 5012.255856] [ 5012.257423] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5012.261613] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5012.264481] do_raw_spin_lock+0x177/0x190 [ 5012.267499] _raw_spin_lock+0x21/0x30 [ 5012.283282] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5012.288202] process_one_work+0x2c8/0x700 [ 5012.298268] worker_thread+0x296/0x6e0 [ 5012.300075] ? rescuer_thread+0x570/0x570 [ 5012.302106] kthread+0x1d1/0x200 [ 5012.307383] ? set_kthread_struct+0x70/0x70 [ 5012.309587] ret_from_fork+0x1f/0x30 [ 5016.052266] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5016.055303] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5016.056207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5016.056288] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5016.056295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5016.056298] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5016.056312] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5016.056317] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5016.056322] RAX: 0000000000000090 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5016.056324] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5016.056327] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5016.056330] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5016.056332] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5016.056336] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5016.056339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5016.056341] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5016.056350] Call Trace: [ 5016.056355] [ 5016.056362] ? show_regs.cold.9+0x22/0x2f [ 5016.056371] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5016.056377] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5016.056382] ? watchdog+0x40/0x40 [ 5016.056386] ? hrtimer_interrupt+0x128/0x2c0 [ 5016.056390] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5016.056398] ? apic_timer_interrupt+0xf/0x20 [ 5016.056403] [ 5016.056406] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5016.056410] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5016.056414] ? dump_stack+0xf6/0x10e [ 5016.056419] do_raw_spin_lock+0x177/0x190 [ 5016.056423] _raw_spin_lock+0x21/0x30 [ 5016.056430] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5016.056726] obd_mod_free+0x46/0x140 [ptlrpc] [ 5016.056938] mdc_commit_open+0x91/0xb0 [mdc] [ 5016.056973] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5016.057183] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5016.057381] after_reply+0xe2c/0x1780 [ptlrpc] [ 5016.057573] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5016.057765] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5016.057988] ? do_wait_intr+0xf0/0xf0 [ 5016.057997] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5016.058204] kthread+0x1d1/0x200 [ 5016.058210] ? set_kthread_struct+0x70/0x70 [ 5016.058215] ret_from_fork+0x1f/0x30 [ 5016.223355] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5016.241994] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5016.247172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5016.251055] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5016.253720] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5016.261725] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5016.265528] RAX: 0000000000001e90 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5016.269201] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5016.274098] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5016.279706] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5016.287279] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5016.291704] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5016.301115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5016.308269] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5016.313222] Call Trace: [ 5016.314345] [ 5016.315648] ? show_regs.cold.9+0x22/0x2f [ 5016.318084] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5016.320711] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5016.325202] ? watchdog+0x40/0x40 [ 5016.329029] ? hrtimer_interrupt+0x128/0x2c0 [ 5016.331407] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5016.336153] ? apic_timer_interrupt+0xf/0x20 [ 5016.338919] [ 5016.342726] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5016.349428] do_raw_spin_lock+0x177/0x190 [ 5016.352399] _raw_spin_lock+0x21/0x30 [ 5016.359182] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5016.364538] ? do_raw_spin_unlock+0x75/0x190 [ 5016.367060] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5016.372309] ? do_wait_intr+0xf0/0xf0 [ 5016.375071] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5016.377826] mdc_close+0x53b/0xd70 [mdc] [ 5016.382285] lmv_close+0x18d/0x490 [lmv] [ 5016.384207] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5016.387467] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5016.390951] ll_md_close+0x274/0x800 [lustre] [ 5016.393562] ll_file_release+0x128/0xc50 [lustre] [ 5016.399800] __fput+0x10d/0x440 [ 5016.403944] ____fput+0x12/0x20 [ 5016.406355] task_work_run+0xc6/0x110 [ 5016.408670] exit_to_usermode_loop+0x1dd/0x1f0 [ 5016.412700] do_syscall_64+0x3d6/0x440 [ 5016.418482] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5016.419786] RIP: 0033:0x7fdc0afbfcd8 [ 5016.424008] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5016.433378] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5016.437616] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5016.441506] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5016.448431] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5016.452733] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5016.457967] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5018.594253] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2993s! [ 5018.600094] Showing busy workqueues and worker pools: [ 5018.601840] workqueue events: flags=0x0 [ 5018.605920] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5018.611902] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5018.617457] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5018.621234] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5018.630550] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5018.643952] pending: kfree_rcu_monitor [ 5018.655141] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5018.666077] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5018.682384] workqueue events_long: flags=0x0 [ 5018.693272] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5018.700873] pending: lu_objects_flush [obdclass] [ 5018.710665] workqueue events_unbound: flags=0x2 [ 5018.720916] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5018.730103] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5018.737741] workqueue events_power_efficient: flags=0x80 [ 5018.743691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5018.752414] pending: neigh_periodic_work, check_lifetime [ 5018.758729] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5018.761850] pending: do_cache_clean, neigh_periodic_work [ 5018.769422] workqueue mm_percpu_wq: flags=0x8 [ 5018.775774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5018.783675] pending: lru_add_drain_per_cpu [ 5018.790273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5018.793113] pending: vmstat_update, lru_add_drain_per_cpu [ 5018.796170] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5018.798771] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5018.801197] workqueue writeback: flags=0x4a [ 5018.802748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5018.806137] pending: wb_workfn, wb_workfn [ 5018.807857] workqueue xprtiod: flags=0xa [ 5018.814302] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5018.818925] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5018.841266] workqueue ipv6_addrconf: flags=0x40008 [ 5018.850287] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5018.855156] pending: addrconf_verify_work [ 5018.863872] workqueue knbd0-recv: flags=0x1a [ 5018.868344] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5018.875289] in-flight: 655:recv_work [ 5018.876487] workqueue ptlrpc_pinger: flags=0x2 [ 5018.883741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5018.889164] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5018.897639] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2993s workers=4 idle: 36396 38 11188 [ 5018.906556] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5009s workers=2 idle: 658 [ 5028.058028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5028.075030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5028.126458] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5028.145441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5028.156065] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5028.162121] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5028.195016] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5028.205945] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5028.208210] RDX: 00000000000005b0 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5028.216997] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5028.228121] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5028.237102] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5028.246246] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5028.250226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5028.255324] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5028.267164] Call Trace: [ 5028.271483] [ 5028.272702] ? show_regs.cold.9+0x22/0x2f [ 5028.276058] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5028.279580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5028.282627] ? watchdog+0x40/0x40 [ 5028.284821] ? hrtimer_interrupt+0x128/0x2c0 [ 5028.289917] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5028.294147] ? apic_timer_interrupt+0xf/0x20 [ 5028.305379] [ 5028.306356] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5028.310921] do_raw_spin_lock+0x177/0x190 [ 5028.314933] _raw_spin_lock+0x21/0x30 [ 5028.316936] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5028.319878] ? do_raw_spin_unlock+0x75/0x190 [ 5028.322716] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5028.330561] ? do_wait_intr+0xf0/0xf0 [ 5028.334880] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5028.342612] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5028.350910] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5028.355246] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5028.360004] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5028.361997] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5028.367319] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5028.373258] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5028.378277] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5028.382924] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5028.386910] lmv_intent_open+0x338/0x1150 [lmv] [ 5028.391361] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5028.395639] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5028.400193] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5028.403672] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5028.407147] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5028.415561] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5028.418174] ? do_raw_spin_unlock+0x75/0x190 [ 5028.422222] ? _raw_spin_unlock+0x12/0x30 [ 5028.426942] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5028.429382] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5028.433979] ll_atomic_open+0x36a/0xd70 [lustre] [ 5028.439311] lookup_open+0x356/0xb40 [ 5028.442655] do_last+0x313/0xfc0 [ 5028.444581] ? nd_jump_root+0xe5/0x160 [ 5028.446418] ? path_init+0x437/0x520 [ 5028.448215] path_openat+0xf7/0x500 [ 5028.451512] ? do_raw_spin_unlock+0x75/0x190 [ 5028.453740] do_filp_open+0x99/0x140 [ 5028.455382] ? getname_flags+0x6e/0x330 [ 5028.462110] ? __check_object_size+0xff/0x256 [ 5028.466126] ? do_raw_spin_unlock+0x75/0x190 [ 5028.468172] ? _raw_spin_unlock+0x12/0x30 [ 5028.472683] do_sys_openat2+0x2b4/0x410 [ 5028.475912] do_sys_open+0x73/0xa0 [ 5028.480284] __x64_sys_openat+0x24/0x30 [ 5028.484085] do_syscall_64+0xc1/0x440 [ 5028.489205] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5028.491287] RIP: 0033:0x7fdc0afbf332 [ 5028.495000] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5028.506995] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5028.511780] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5028.515044] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5028.518347] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5028.522145] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5028.524606] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5040.051019] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5040.054509] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5040.076914] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5040.081527] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5040.085182] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5040.088744] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5040.091166] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5040.100590] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5040.105843] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5040.108627] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5040.113177] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5040.120007] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5040.125487] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5040.129427] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5040.134538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5040.138159] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5040.142291] Call Trace: [ 5040.143541] [ 5040.144666] ? show_regs.cold.9+0x22/0x2f [ 5040.147075] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5040.148922] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5040.150871] ? watchdog+0x40/0x40 [ 5040.152687] ? hrtimer_interrupt+0x128/0x2c0 [ 5040.155619] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5040.158033] ? apic_timer_interrupt+0xf/0x20 [ 5040.162399] [ 5040.165054] ? native_safe_halt+0x2e/0x50 [ 5040.166677] kvm_wait+0xcd/0xe0 [ 5040.167860] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5040.170667] do_raw_spin_lock+0x177/0x190 [ 5040.176543] _raw_spin_lock+0x21/0x30 [ 5040.178397] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5040.181447] process_one_work+0x2c8/0x700 [ 5040.183590] worker_thread+0x296/0x6e0 [ 5040.184969] ? rescuer_thread+0x570/0x570 [ 5040.186427] kthread+0x1d1/0x200 [ 5040.189000] ? set_kthread_struct+0x70/0x70 [ 5040.190625] ret_from_fork+0x1f/0x30 [ 5044.048009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5044.054032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5044.056251] Modules linked in: [ 5044.059586] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5044.059663] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5044.059671] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5044.059674] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5044.059687] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5044.059691] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5044.059696] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5044.059698] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5044.059700] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5044.059702] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5044.059705] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5044.059708] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5044.059711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5044.059714] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5044.059722] Call Trace: [ 5044.059728] [ 5044.059734] ? show_regs.cold.9+0x22/0x2f [ 5044.059744] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5044.059750] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5044.059755] ? watchdog+0x40/0x40 [ 5044.059759] ? hrtimer_interrupt+0x128/0x2c0 [ 5044.059763] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5044.059771] ? apic_timer_interrupt+0xf/0x20 [ 5044.059776] [ 5044.059779] ? native_safe_halt+0x2e/0x50 [ 5044.059783] kvm_wait+0xcd/0xe0 [ 5044.059791] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5044.059799] do_raw_spin_lock+0x177/0x190 [ 5044.059803] _raw_spin_lock+0x21/0x30 [ 5044.059808] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5044.060088] ? do_raw_spin_unlock+0x75/0x190 [ 5044.060094] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5044.061569] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 5044.092687] ? do_wait_intr+0xf0/0xf0 [ 5044.111564] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5044.111626] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5044.111633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5044.111636] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5044.111650] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5044.111654] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5044.111658] RAX: 00000000000000b1 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5044.111661] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5044.111663] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5044.111665] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5044.111668] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5044.111671] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5044.111673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5044.111676] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5044.111684] Call Trace: [ 5044.111690] [ 5044.111695] ? show_regs.cold.9+0x22/0x2f [ 5044.111705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5044.111711] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5044.111716] ? watchdog+0x40/0x40 [ 5044.111720] ? hrtimer_interrupt+0x128/0x2c0 [ 5044.111725] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5044.111735] ? apic_timer_interrupt+0xf/0x20 [ 5044.111739] [ 5044.111742] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5044.111747] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5044.111751] ? dump_stack+0xf6/0x10e [ 5044.111756] do_raw_spin_lock+0x177/0x190 [ 5044.111761] _raw_spin_lock+0x21/0x30 [ 5044.111768] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5044.112039] obd_mod_free+0x46/0x140 [ptlrpc] [ 5044.112242] mdc_commit_open+0x91/0xb0 [mdc] [ 5044.112282] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5044.112479] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5044.112695] after_reply+0xe2c/0x1780 [ptlrpc] [ 5044.112913] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5044.113135] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5044.113335] ? do_wait_intr+0xf0/0xf0 [ 5044.113345] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5044.113555] kthread+0x1d1/0x200 [ 5044.113561] ? set_kthread_struct+0x70/0x70 [ 5044.113565] ret_from_fork+0x1f/0x30 [ 5044.773107] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5044.777204] mdc_close+0x53b/0xd70 [mdc] [ 5044.779062] lmv_close+0x18d/0x490 [lmv] [ 5044.786709] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5044.795092] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5044.804140] ll_md_close+0x274/0x800 [lustre] [ 5044.810606] ll_file_release+0x128/0xc50 [lustre] [ 5044.816667] __fput+0x10d/0x440 [ 5044.818951] ____fput+0x12/0x20 [ 5044.825050] task_work_run+0xc6/0x110 [ 5044.831025] exit_to_usermode_loop+0x1dd/0x1f0 [ 5044.840968] do_syscall_64+0x3d6/0x440 [ 5044.842437] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5044.850032] RIP: 0033:0x7fdc0afbfcd8 [ 5044.852813] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5044.879576] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5044.903012] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5044.921828] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5044.938361] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5044.954665] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5044.963068] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5049.311858] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3024s! [ 5049.317158] Showing busy workqueues and worker pools: [ 5049.327053] workqueue events: flags=0x0 [ 5049.328343] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5049.337921] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5049.344651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5049.358460] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5049.370942] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5049.386890] pending: kfree_rcu_monitor [ 5049.398387] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5049.411588] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5049.430425] workqueue events_long: flags=0x0 [ 5049.442497] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5049.456175] pending: lu_objects_flush [obdclass] [ 5049.466318] workqueue events_unbound: flags=0x2 [ 5049.477582] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5049.490585] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5049.508913] workqueue events_power_efficient: flags=0x80 [ 5049.517384] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5049.531459] pending: neigh_periodic_work, check_lifetime [ 5049.534906] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5049.545955] pending: do_cache_clean, neigh_periodic_work [ 5049.555748] workqueue mm_percpu_wq: flags=0x8 [ 5049.568676] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5049.579745] pending: lru_add_drain_per_cpu [ 5049.588676] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5049.596932] pending: vmstat_update, lru_add_drain_per_cpu [ 5049.602500] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5049.615753] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5049.623572] workqueue writeback: flags=0x4a [ 5049.636628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5049.648363] pending: wb_workfn, wb_workfn [ 5049.652208] workqueue xprtiod: flags=0xa [ 5049.658616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5049.672265] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5049.684618] workqueue ipv6_addrconf: flags=0x40008 [ 5049.692229] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5049.710626] pending: addrconf_verify_work [ 5049.713426] workqueue knbd0-recv: flags=0x1a [ 5049.724618] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5049.739417] in-flight: 655:recv_work [ 5049.745172] workqueue ptlrpc_pinger: flags=0x2 [ 5049.754587] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5049.767266] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5049.782126] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3024s workers=4 idle: 36396 38 11188 [ 5049.789977] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5040s workers=2 idle: 658 [ 5056.052028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5056.056896] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5056.084095] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5056.090569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5056.099347] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5056.102427] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5056.112556] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5056.115685] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5056.118841] RDX: 00000000000007b4 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5056.122513] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5056.125784] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5056.129415] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5056.133525] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5056.136492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5056.138712] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5056.140956] Call Trace: [ 5056.142087] [ 5056.142838] ? show_regs.cold.9+0x22/0x2f [ 5056.144089] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5056.146708] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5056.148428] ? watchdog+0x40/0x40 [ 5056.149769] ? hrtimer_interrupt+0x128/0x2c0 [ 5056.151455] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5056.152912] ? apic_timer_interrupt+0xf/0x20 [ 5056.155053] [ 5056.156090] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5056.158363] do_raw_spin_lock+0x177/0x190 [ 5056.160058] _raw_spin_lock+0x21/0x30 [ 5056.161399] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5056.163614] ? do_raw_spin_unlock+0x75/0x190 [ 5056.165740] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5056.167499] ? do_wait_intr+0xf0/0xf0 [ 5056.168959] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5056.171064] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5056.172894] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5056.175420] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5056.177657] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5056.179822] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5056.182267] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5056.183815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5056.185840] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5056.188195] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5056.190250] lmv_intent_open+0x338/0x1150 [lmv] [ 5056.191760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5056.193904] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5056.196591] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5056.198497] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5056.200360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5056.202214] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5056.205447] ? do_raw_spin_unlock+0x75/0x190 [ 5056.208199] ? _raw_spin_unlock+0x12/0x30 [ 5056.209735] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5056.211891] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5056.214380] ll_atomic_open+0x36a/0xd70 [lustre] [ 5056.216529] lookup_open+0x356/0xb40 [ 5056.218016] do_last+0x313/0xfc0 [ 5056.219765] ? nd_jump_root+0xe5/0x160 [ 5056.221555] ? path_init+0x437/0x520 [ 5056.223131] path_openat+0xf7/0x500 [ 5056.224654] ? do_raw_spin_unlock+0x75/0x190 [ 5056.226614] do_filp_open+0x99/0x140 [ 5056.227873] ? getname_flags+0x6e/0x330 [ 5056.230209] ? __check_object_size+0xff/0x256 [ 5056.232392] ? do_raw_spin_unlock+0x75/0x190 [ 5056.234474] ? _raw_spin_unlock+0x12/0x30 [ 5056.236061] do_sys_openat2+0x2b4/0x410 [ 5056.237906] do_sys_open+0x73/0xa0 [ 5056.239380] __x64_sys_openat+0x24/0x30 [ 5056.240793] do_syscall_64+0xc1/0x440 [ 5056.242289] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5056.244885] RIP: 0033:0x7fdc0afbf332 [ 5056.246529] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5056.253414] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5056.256349] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5056.259262] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5056.261981] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5056.264446] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5056.267195] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5068.051009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5068.064299] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5068.104860] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5068.114466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5068.120462] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5068.125667] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5068.130664] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5068.147852] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5068.151035] RAX: 00000000000074f2 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5068.154577] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5068.158190] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5068.161410] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5068.164893] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5068.167849] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5068.171627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5068.175952] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5068.181323] Call Trace: [ 5068.182397] [ 5068.183794] ? show_regs.cold.9+0x22/0x2f [ 5068.185452] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5068.188082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5068.190357] ? watchdog+0x40/0x40 [ 5068.193188] ? hrtimer_interrupt+0x128/0x2c0 [ 5068.195133] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5068.205374] ? apic_timer_interrupt+0xf/0x20 [ 5068.208911] [ 5068.212020] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5068.214004] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5068.221923] do_raw_spin_lock+0x177/0x190 [ 5068.226405] _raw_spin_lock+0x21/0x30 [ 5068.229035] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5068.233033] process_one_work+0x2c8/0x700 [ 5068.234984] worker_thread+0x296/0x6e0 [ 5068.240196] ? rescuer_thread+0x570/0x570 [ 5068.242209] kthread+0x1d1/0x200 [ 5068.249128] ? set_kthread_struct+0x70/0x70 [ 5068.252023] ret_from_fork+0x1f/0x30 [ 5072.054526] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5072.058993] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5072.089503] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5072.106398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5072.124023] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5072.136553] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5072.153000] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5072.161985] RAX: 0000000000004b2a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5072.164906] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5072.170429] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5072.176285] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5072.180707] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5072.186354] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5072.192636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5072.194371] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5072.196723] Call Trace: [ 5072.198022] [ 5072.200060] ? show_regs.cold.9+0x22/0x2f [ 5072.201055] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5072.202373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5072.209696] ? watchdog+0x40/0x40 [ 5072.210615] ? hrtimer_interrupt+0x128/0x2c0 [ 5072.211726] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5072.215071] ? apic_timer_interrupt+0xf/0x20 [ 5072.220367] [ 5072.220910] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5072.222916] do_raw_spin_lock+0x177/0x190 [ 5072.223981] _raw_spin_lock+0x21/0x30 [ 5072.224942] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5072.228609] ? do_raw_spin_unlock+0x75/0x190 [ 5072.230331] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5072.232407] ? do_wait_intr+0xf0/0xf0 [ 5072.234167] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5072.236483] mdc_close+0x53b/0xd70 [mdc] [ 5072.238030] lmv_close+0x18d/0x490 [lmv] [ 5072.240324] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5072.242862] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5072.250185] ll_md_close+0x274/0x800 [lustre] [ 5072.257306] ll_file_release+0x128/0xc50 [lustre] [ 5072.261053] __fput+0x10d/0x440 [ 5072.262157] ____fput+0x12/0x20 [ 5072.265056] task_work_run+0xc6/0x110 [ 5072.266472] exit_to_usermode_loop+0x1dd/0x1f0 [ 5072.269057] do_syscall_64+0x3d6/0x440 [ 5072.271024] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5072.276858] RIP: 0033:0x7fdc0afbfcd8 [ 5072.280399] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5072.291587] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5072.298532] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5072.305534] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5072.316278] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5072.322604] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5072.327033] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5076.049284] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5076.057060] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5076.089177] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5076.099750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5076.106933] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5076.111246] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5076.141822] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5076.144510] RAX: 0000000000002259 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5076.154858] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5076.165304] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5076.172081] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5076.186250] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5076.199177] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5076.202954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5076.212077] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5076.215593] Call Trace: [ 5076.216506] [ 5076.222904] ? show_regs.cold.9+0x22/0x2f [ 5076.224960] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5076.227074] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5076.231187] ? watchdog+0x40/0x40 [ 5076.234838] ? hrtimer_interrupt+0x128/0x2c0 [ 5076.236381] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5076.243385] ? apic_timer_interrupt+0xf/0x20 [ 5076.246381] [ 5076.247910] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5076.250765] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5076.253463] ? dump_stack+0xf6/0x10e [ 5076.254752] do_raw_spin_lock+0x177/0x190 [ 5076.259642] _raw_spin_lock+0x21/0x30 [ 5076.272370] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5076.277734] obd_mod_free+0x46/0x140 [ptlrpc] [ 5076.287647] mdc_commit_open+0x91/0xb0 [mdc] [ 5076.290285] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5076.292362] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5076.296509] after_reply+0xe2c/0x1780 [ptlrpc] [ 5076.301040] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5076.306929] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5076.312538] ? do_wait_intr+0xf0/0xf0 [ 5076.317355] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5076.321232] kthread+0x1d1/0x200 [ 5076.322627] ? set_kthread_struct+0x70/0x70 [ 5076.324907] ret_from_fork+0x1f/0x30 [ 5080.033331] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3055s! [ 5080.038698] Showing busy workqueues and worker pools: [ 5080.041649] workqueue events: flags=0x0 [ 5080.044135] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5080.048111] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5080.051750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5080.055438] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5080.060958] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5080.065060] pending: kfree_rcu_monitor [ 5080.067605] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5080.071599] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5080.080077] workqueue events_long: flags=0x0 [ 5080.082921] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5080.086571] pending: lu_objects_flush [obdclass] [ 5080.089471] workqueue events_unbound: flags=0x2 [ 5080.092122] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5080.096002] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5080.102729] workqueue events_power_efficient: flags=0x80 [ 5080.105763] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5080.108852] pending: neigh_periodic_work, check_lifetime [ 5080.111094] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5080.113789] pending: do_cache_clean, neigh_periodic_work [ 5080.115968] workqueue mm_percpu_wq: flags=0x8 [ 5080.117834] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5080.120747] pending: lru_add_drain_per_cpu [ 5080.123776] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5080.127129] pending: vmstat_update, lru_add_drain_per_cpu [ 5080.135010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5080.139477] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5080.142847] workqueue writeback: flags=0x4a [ 5080.146843] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5080.149492] pending: wb_workfn, wb_workfn [ 5080.152172] workqueue xprtiod: flags=0xa [ 5080.154179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5080.157378] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5080.161337] workqueue ipv6_addrconf: flags=0x40008 [ 5080.164323] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5080.167559] pending: addrconf_verify_work [ 5080.169560] workqueue knbd0-recv: flags=0x1a [ 5080.171595] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5080.175261] in-flight: 655:recv_work [ 5080.177079] workqueue ptlrpc_pinger: flags=0x2 [ 5080.179133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5080.183218] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5080.186514] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3055s workers=4 idle: 36396 38 11188 [ 5080.191257] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5070s workers=2 idle: 658 [ 5084.052081] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5084.055939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5084.083051] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5084.089518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5084.095071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5084.097980] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5084.109383] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5084.113453] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5084.117910] RDX: 0000000000004ef6 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5084.121947] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5084.126211] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5084.130354] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5084.135102] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5084.139782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5084.143519] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5084.146147] Call Trace: [ 5084.146998] [ 5084.148608] ? show_regs.cold.9+0x22/0x2f [ 5084.151048] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5084.153508] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5084.155986] ? watchdog+0x40/0x40 [ 5084.157504] ? hrtimer_interrupt+0x128/0x2c0 [ 5084.159480] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5084.161987] ? apic_timer_interrupt+0xf/0x20 [ 5084.164166] [ 5084.165394] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5084.168356] do_raw_spin_lock+0x177/0x190 [ 5084.170871] _raw_spin_lock+0x21/0x30 [ 5084.172717] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5084.175862] ? do_raw_spin_unlock+0x75/0x190 [ 5084.178347] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5084.181232] ? do_wait_intr+0xf0/0xf0 [ 5084.183250] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5084.186645] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5084.189650] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5084.192772] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5084.195412] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5084.198440] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5084.201004] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5084.203142] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5084.205727] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5084.208638] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5084.211268] lmv_intent_open+0x338/0x1150 [lmv] [ 5084.213665] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5084.216926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5084.219223] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5084.221178] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5084.223451] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5084.225980] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5084.228198] ? do_raw_spin_unlock+0x75/0x190 [ 5084.230432] ? _raw_spin_unlock+0x12/0x30 [ 5084.232005] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5084.234322] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5084.236654] ll_atomic_open+0x36a/0xd70 [lustre] [ 5084.238627] lookup_open+0x356/0xb40 [ 5084.240361] do_last+0x313/0xfc0 [ 5084.241615] ? nd_jump_root+0xe5/0x160 [ 5084.243104] ? path_init+0x437/0x520 [ 5084.244270] path_openat+0xf7/0x500 [ 5084.245098] ? do_raw_spin_unlock+0x75/0x190 [ 5084.246159] do_filp_open+0x99/0x140 [ 5084.247518] ? getname_flags+0x6e/0x330 [ 5084.248931] ? __check_object_size+0xff/0x256 [ 5084.250448] ? do_raw_spin_unlock+0x75/0x190 [ 5084.252132] ? _raw_spin_unlock+0x12/0x30 [ 5084.253855] do_sys_openat2+0x2b4/0x410 [ 5084.255485] do_sys_open+0x73/0xa0 [ 5084.256844] __x64_sys_openat+0x24/0x30 [ 5084.258432] do_syscall_64+0xc1/0x440 [ 5084.259965] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5084.261815] RIP: 0033:0x7fdc0afbf332 [ 5084.262728] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5084.269337] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5084.272100] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5084.274193] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5084.276398] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5084.279720] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5084.282739] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5096.051029] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 5096.054008] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5096.073101] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5096.077944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5096.081857] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5096.084979] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5096.086946] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5096.093415] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5096.096113] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5096.099016] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5096.102164] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5096.104722] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5096.107476] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5096.110302] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5096.113292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5096.115589] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5096.118757] Call Trace: [ 5096.119907] [ 5096.120750] ? show_regs.cold.9+0x22/0x2f [ 5096.122480] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5096.124214] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5096.126069] ? watchdog+0x40/0x40 [ 5096.127172] ? hrtimer_interrupt+0x128/0x2c0 [ 5096.128791] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5096.130665] ? apic_timer_interrupt+0xf/0x20 [ 5096.132538] [ 5096.133426] ? native_safe_halt+0x2e/0x50 [ 5096.135146] kvm_wait+0xcd/0xe0 [ 5096.136778] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5096.139052] do_raw_spin_lock+0x177/0x190 [ 5096.140582] _raw_spin_lock+0x21/0x30 [ 5096.142252] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5096.144743] process_one_work+0x2c8/0x700 [ 5096.146669] worker_thread+0x296/0x6e0 [ 5096.148330] ? rescuer_thread+0x570/0x570 [ 5096.150037] kthread+0x1d1/0x200 [ 5096.151576] ? set_kthread_struct+0x70/0x70 [ 5096.153616] ret_from_fork+0x1f/0x30 [ 5100.054164] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5100.057532] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5100.075512] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5100.080920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5100.084823] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5100.087255] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5100.095432] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5100.099973] RAX: 00000000000040cf RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5100.104264] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5100.107867] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5100.113865] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5100.121477] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5100.124278] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5100.129023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5100.134215] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5100.137992] Call Trace: [ 5100.139970] [ 5100.141717] ? show_regs.cold.9+0x22/0x2f [ 5100.143960] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5100.147809] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5100.149988] ? watchdog+0x40/0x40 [ 5100.152834] ? hrtimer_interrupt+0x128/0x2c0 [ 5100.156309] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5100.159439] ? apic_timer_interrupt+0xf/0x20 [ 5100.161292] [ 5100.162098] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5100.166048] do_raw_spin_lock+0x177/0x190 [ 5100.167845] _raw_spin_lock+0x21/0x30 [ 5100.169271] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5100.171329] ? do_raw_spin_unlock+0x75/0x190 [ 5100.172740] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5100.175415] ? do_wait_intr+0xf0/0xf0 [ 5100.179520] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5100.181886] mdc_close+0x53b/0xd70 [mdc] [ 5100.183315] lmv_close+0x18d/0x490 [lmv] [ 5100.185539] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5100.188057] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5100.189933] ll_md_close+0x274/0x800 [lustre] [ 5100.192690] ll_file_release+0x128/0xc50 [lustre] [ 5100.196423] __fput+0x10d/0x440 [ 5100.198689] ____fput+0x12/0x20 [ 5100.201400] task_work_run+0xc6/0x110 [ 5100.203314] exit_to_usermode_loop+0x1dd/0x1f0 [ 5100.205424] do_syscall_64+0x3d6/0x440 [ 5100.206896] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5100.211308] RIP: 0033:0x7fdc0afbfcd8 [ 5100.214087] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5100.222504] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5100.224948] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5100.229380] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5100.232459] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5100.235337] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5100.237803] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5108.048011] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 5108.051765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5108.073861] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5108.079066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5108.083433] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5108.085918] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5108.094963] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5108.098724] RAX: 0000000000005225 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5108.102300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5108.105747] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5108.109459] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5108.113350] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5108.117105] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5108.121074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5108.123774] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5108.127066] Call Trace: [ 5108.128334] [ 5108.129455] ? show_regs.cold.9+0x22/0x2f [ 5108.131835] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5108.133866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5108.136193] ? watchdog+0x40/0x40 [ 5108.138073] ? hrtimer_interrupt+0x128/0x2c0 [ 5108.140290] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5108.142724] ? apic_timer_interrupt+0xf/0x20 [ 5108.144751] [ 5108.145702] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5108.147929] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5108.150575] ? dump_stack+0xf6/0x10e [ 5108.152348] do_raw_spin_lock+0x177/0x190 [ 5108.154427] _raw_spin_lock+0x21/0x30 [ 5108.156100] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5108.159463] obd_mod_free+0x46/0x140 [ptlrpc] [ 5108.161727] mdc_commit_open+0x91/0xb0 [mdc] [ 5108.164295] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5108.167046] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5108.169943] after_reply+0xe2c/0x1780 [ptlrpc] [ 5108.172491] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5108.175758] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5108.178251] ? do_wait_intr+0xf0/0xf0 [ 5108.179943] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5108.182577] kthread+0x1d1/0x200 [ 5108.184472] ? set_kthread_struct+0x70/0x70 [ 5108.186400] ret_from_fork+0x1f/0x30 [ 5110.239048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3085s! [ 5110.242679] Showing busy workqueues and worker pools: [ 5110.245274] workqueue events: flags=0x0 [ 5110.247500] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5110.250917] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5110.253481] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5110.255876] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5110.260714] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5110.263334] pending: kfree_rcu_monitor [ 5110.265527] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5110.268250] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5110.273249] workqueue events_long: flags=0x0 [ 5110.274890] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5110.277931] pending: lu_objects_flush [obdclass] [ 5110.279796] workqueue events_unbound: flags=0x2 [ 5110.281125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5110.283213] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5110.286819] workqueue events_power_efficient: flags=0x80 [ 5110.288254] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5110.290174] pending: neigh_periodic_work, check_lifetime [ 5110.292123] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5110.294355] pending: do_cache_clean, neigh_periodic_work [ 5110.296169] workqueue mm_percpu_wq: flags=0x8 [ 5110.297295] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5110.299290] pending: lru_add_drain_per_cpu [ 5110.301250] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5110.304037] pending: vmstat_update, lru_add_drain_per_cpu [ 5110.306077] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5110.308118] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5110.310539] workqueue writeback: flags=0x4a [ 5110.311939] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5110.314438] pending: wb_workfn, wb_workfn [ 5110.315730] workqueue xprtiod: flags=0xa [ 5110.317710] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5110.321144] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5110.325352] workqueue ipv6_addrconf: flags=0x40008 [ 5110.327756] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5110.331027] pending: addrconf_verify_work [ 5110.333172] workqueue knbd0-recv: flags=0x1a [ 5110.335335] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5110.339132] in-flight: 655:recv_work [ 5110.341012] workqueue ptlrpc_pinger: flags=0x2 [ 5110.343086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5110.345524] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5110.348486] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3085s workers=4 idle: 36396 38 11188 [ 5110.352684] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5100s workers=2 idle: 658 [ 5112.055982] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5112.063507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5112.092543] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5112.099661] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5112.105857] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5112.109970] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5112.122079] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5112.126813] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5112.131384] RDX: 00000000000061af RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5112.135867] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5112.140156] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5112.145078] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5112.149830] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5112.154777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5112.158705] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5112.163485] Call Trace: [ 5112.164760] [ 5112.166106] ? show_regs.cold.9+0x22/0x2f [ 5112.168857] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5112.171630] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5112.174892] ? watchdog+0x40/0x40 [ 5112.177839] ? hrtimer_interrupt+0x128/0x2c0 [ 5112.180034] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5112.182916] ? apic_timer_interrupt+0xf/0x20 [ 5112.185747] [ 5112.187488] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5112.189970] do_raw_spin_lock+0x177/0x190 [ 5112.192392] _raw_spin_lock+0x21/0x30 [ 5112.195720] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5112.199567] ? do_raw_spin_unlock+0x75/0x190 [ 5112.203055] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5112.207867] ? do_wait_intr+0xf0/0xf0 [ 5112.210150] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5112.214191] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5112.218292] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5112.221970] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5112.225454] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5112.228805] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5112.231879] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5112.234138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5112.237060] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5112.240262] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5112.242825] lmv_intent_open+0x338/0x1150 [lmv] [ 5112.244745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5112.246998] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5112.249297] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5112.251441] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5112.253796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5112.256917] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5112.258958] ? do_raw_spin_unlock+0x75/0x190 [ 5112.261031] ? _raw_spin_unlock+0x12/0x30 [ 5112.262777] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5112.264665] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5112.266379] ll_atomic_open+0x36a/0xd70 [lustre] [ 5112.268026] lookup_open+0x356/0xb40 [ 5112.269908] do_last+0x313/0xfc0 [ 5112.271065] ? nd_jump_root+0xe5/0x160 [ 5112.272434] ? path_init+0x437/0x520 [ 5112.273812] path_openat+0xf7/0x500 [ 5112.275147] ? do_raw_spin_unlock+0x75/0x190 [ 5112.276754] do_filp_open+0x99/0x140 [ 5112.277867] ? getname_flags+0x6e/0x330 [ 5112.279101] ? __check_object_size+0xff/0x256 [ 5112.280536] ? do_raw_spin_unlock+0x75/0x190 [ 5112.282103] ? _raw_spin_unlock+0x12/0x30 [ 5112.283425] do_sys_openat2+0x2b4/0x410 [ 5112.284799] do_sys_open+0x73/0xa0 [ 5112.286171] __x64_sys_openat+0x24/0x30 [ 5112.287388] do_syscall_64+0xc1/0x440 [ 5112.288495] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5112.290053] RIP: 0033:0x7fdc0afbf332 [ 5112.291266] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5112.297328] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5112.299873] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5112.302474] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5112.305289] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5112.308137] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5112.311015] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5124.051172] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 5124.058163] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5124.100043] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5124.116548] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5124.122965] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5124.131197] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5124.134392] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5124.152843] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5124.158563] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5124.163154] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5124.169129] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5124.171705] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5124.175303] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5124.181631] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5124.188530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5124.191232] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5124.195718] Call Trace: [ 5124.197248] [ 5124.198604] ? show_regs.cold.9+0x22/0x2f [ 5124.200699] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5124.202677] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5124.204514] ? watchdog+0x40/0x40 [ 5124.205692] ? hrtimer_interrupt+0x128/0x2c0 [ 5124.207299] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5124.209019] ? apic_timer_interrupt+0xf/0x20 [ 5124.210571] [ 5124.211288] ? native_safe_halt+0x2e/0x50 [ 5124.212659] kvm_wait+0xcd/0xe0 [ 5124.213830] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5124.215605] do_raw_spin_lock+0x177/0x190 [ 5124.217160] _raw_spin_lock+0x21/0x30 [ 5124.218477] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5124.220499] process_one_work+0x2c8/0x700 [ 5124.221805] worker_thread+0x296/0x6e0 [ 5124.223208] ? rescuer_thread+0x570/0x570 [ 5124.224568] kthread+0x1d1/0x200 [ 5124.225626] ? set_kthread_struct+0x70/0x70 [ 5124.227037] ret_from_fork+0x1f/0x30 [ 5128.056593] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5128.060181] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5128.079110] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5128.083770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5128.087400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5128.090017] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5128.098160] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5128.107286] RAX: 000000000000120a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5128.117878] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5128.121354] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5128.124630] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5128.128573] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5128.133570] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5128.139076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5128.159164] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5128.163722] Call Trace: [ 5128.164617] [ 5128.167511] ? show_regs.cold.9+0x22/0x2f [ 5128.171265] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5128.175214] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5128.177924] ? watchdog+0x40/0x40 [ 5128.179244] ? hrtimer_interrupt+0x128/0x2c0 [ 5128.182275] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5128.188309] ? apic_timer_interrupt+0xf/0x20 [ 5128.190558] [ 5128.191822] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5128.193985] do_raw_spin_lock+0x177/0x190 [ 5128.196237] _raw_spin_lock+0x21/0x30 [ 5128.197919] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5128.202267] ? do_raw_spin_unlock+0x75/0x190 [ 5128.207275] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5128.212609] ? do_wait_intr+0xf0/0xf0 [ 5128.222440] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5128.231031] mdc_close+0x53b/0xd70 [mdc] [ 5128.234617] lmv_close+0x18d/0x490 [lmv] [ 5128.239856] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5128.253320] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5128.257544] ll_md_close+0x274/0x800 [lustre] [ 5128.263636] ll_file_release+0x128/0xc50 [lustre] [ 5128.270200] __fput+0x10d/0x440 [ 5128.272292] ____fput+0x12/0x20 [ 5128.275921] task_work_run+0xc6/0x110 [ 5128.282786] exit_to_usermode_loop+0x1dd/0x1f0 [ 5128.285266] do_syscall_64+0x3d6/0x440 [ 5128.291981] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5128.304524] RIP: 0033:0x7fdc0afbfcd8 [ 5128.310357] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5128.330497] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5128.340501] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5128.346960] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5128.358306] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5128.370897] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5128.383631] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5136.048207] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 5136.053873] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5136.083228] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5136.089397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5136.095195] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5136.097064] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5136.106346] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5136.109514] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5136.112700] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5136.115333] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5136.117623] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5136.122835] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5136.125960] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5136.129962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5136.132392] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5136.135999] Call Trace: [ 5136.136999] [ 5136.137774] ? show_regs.cold.9+0x22/0x2f [ 5136.145782] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5136.152445] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5136.156315] ? watchdog+0x40/0x40 [ 5136.158181] ? hrtimer_interrupt+0x128/0x2c0 [ 5136.161368] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5136.165101] ? apic_timer_interrupt+0xf/0x20 [ 5136.167022] [ 5136.168745] ? native_safe_halt+0x2e/0x50 [ 5136.170090] kvm_wait+0xcd/0xe0 [ 5136.171808] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5136.173900] ? dump_stack+0xf6/0x10e [ 5136.176060] do_raw_spin_lock+0x177/0x190 [ 5136.178167] _raw_spin_lock+0x21/0x30 [ 5136.180089] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5136.182838] obd_mod_free+0x46/0x140 [ptlrpc] [ 5136.185010] mdc_commit_open+0x91/0xb0 [mdc] [ 5136.187075] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5136.189657] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5136.192568] after_reply+0xe2c/0x1780 [ptlrpc] [ 5136.194787] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5136.197729] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5136.200047] ? do_wait_intr+0xf0/0xf0 [ 5136.201642] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5136.204068] kthread+0x1d1/0x200 [ 5136.206099] ? set_kthread_struct+0x70/0x70 [ 5136.208603] ret_from_fork+0x1f/0x30 [ 5139.413936] rcu: INFO: rcu_sched self-detected stall on CPU [ 5139.417062] rcu: 0-...!: (1860184 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 5139.428401] (t=3120070 jiffies g=270761 q=3452) [ 5139.433240] rcu: rcu_sched kthread starved for 3115002 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 5139.443310] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5139.452782] rcu: RCU grace-period kthread stack dump: [ 5139.457088] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5139.467739] Call Trace: [ 5139.469433] __schedule+0x351/0xcb0 [ 5139.471666] schedule+0xc0/0x180 [ 5139.473713] schedule_timeout+0xb4/0x190 [ 5139.478688] ? do_raw_spin_unlock+0x75/0x190 [ 5139.482363] ? __next_timer_interrupt+0x160/0x160 [ 5139.485192] ? prepare_to_swait_event+0xf3/0x140 [ 5139.490143] rcu_gp_kthread+0x8dd/0x15b0 [ 5139.492804] ? force_qs_rnp+0x2c0/0x2c0 [ 5139.494817] kthread+0x1d1/0x200 [ 5139.501654] ? set_kthread_struct+0x70/0x70 [ 5139.505667] ret_from_fork+0x1f/0x30 [ 5139.512378] rcu: Stack dump where RCU GP kthread last ran: [ 5139.519054] Sending NMI from CPU 0 to CPUs 2: [ 5139.525692] NMI backtrace for cpu 2 [ 5139.525703] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5139.525705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5139.525707] RIP: 0010:rb_insert_color+0x3/0x240 [ 5139.525708] Code: 48 83 05 2f 30 ce 02 01 e9 4b ff ff ff 48 83 05 3a 30 ce 02 01 4c 89 78 10 eb 9a 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 8b 07 <48> 85 c0 0f 84 cf 01 00 00 48 8b 10 f6 c2 01 0f 85 aa 01 00 00 48 [ 5139.525710] RSP: 0018:ffffba8e40d58ed8 EFLAGS: 00000006 [ 5139.525713] RAX: ffff9fe642125760 RBX: 0000000000000001 RCX: ffff9fe642125770 [ 5139.525714] RDX: 0000000000000000 RSI: ffff9fe642124ee0 RDI: ffff9fe642125540 [ 5139.525715] RBP: ffff9fe642124e40 R08: 000004aca37156c0 R09: 000004aca37156c0 [ 5139.525716] R10: 00000000000f4240 R11: 000004aca37156c0 R12: ffff9fe642125540 [ 5139.525717] R13: 0000000000000000 R14: ffffffff934c8b00 R15: ffff9fe642124fb8 [ 5139.525719] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5139.525720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5139.526005] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5139.526008] Call Trace: [ 5139.526009] [ 5139.526010] ? show_regs.cold.9+0x22/0x2f [ 5139.526011] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5139.526012] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5139.526013] ? nmi_handle+0xbd/0x210 [ 5139.526014] ? default_do_nmi+0x4b/0x1b0 [ 5139.526015] ? do_nmi+0x280/0x380 [ 5139.526016] ? end_repeat_nmi+0x16/0x1f [ 5139.526017] ? rb_insert_color+0x3/0x240 [ 5139.526019] ? rb_insert_color+0x3/0x240 [ 5139.526020] ? rb_insert_color+0x3/0x240 [ 5139.526021] [ 5139.526022] [ 5139.526023] timerqueue_add+0x7f/0xb0 [ 5139.526024] enqueue_hrtimer+0x77/0x130 [ 5139.526025] __hrtimer_run_queues+0x23a/0x4d0 [ 5139.526026] hrtimer_interrupt+0x128/0x2c0 [ 5139.526027] smp_apic_timer_interrupt+0xa6/0x270 [ 5139.526033] apic_timer_interrupt+0xf/0x20 [ 5139.526034] [ 5139.526035] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5139.526036] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5139.526039] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5139.526044] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5139.526046] RDX: 0000000000005136 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5139.526047] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5139.526048] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5139.526049] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5139.526053] do_raw_spin_lock+0x177/0x190 [ 5139.526054] _raw_spin_lock+0x21/0x30 [ 5139.526055] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5139.526056] ? do_raw_spin_unlock+0x75/0x190 [ 5139.526058] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5139.526059] ? do_wait_intr+0xf0/0xf0 [ 5139.526060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5139.526061] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5139.526062] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5139.526063] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5139.526064] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5139.526066] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5139.526067] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5139.526070] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5139.526072] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5139.526073] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5139.526074] lmv_intent_open+0x338/0x1150 [lmv] [ 5139.526075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5139.526076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5139.526078] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5139.526079] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5139.526082] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5139.526084] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5139.526085] ? do_raw_spin_unlock+0x75/0x190 [ 5139.526086] ? _raw_spin_unlock+0x12/0x30 [ 5139.526087] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5139.526090] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5139.526092] ll_atomic_open+0x36a/0xd70 [lustre] [ 5139.526093] lookup_open+0x356/0xb40 [ 5139.526094] do_last+0x313/0xfc0 [ 5139.526095] ? nd_jump_root+0xe5/0x160 [ 5139.526096] ? path_init+0x437/0x520 [ 5139.526097] path_openat+0xf7/0x500 [ 5139.526098] ? do_raw_spin_unlock+0x75/0x190 [ 5139.526099] do_filp_open+0x99/0x140 [ 5139.526100] ? getname_flags+0x6e/0x330 [ 5139.526104] ? __check_object_size+0xff/0x256 [ 5139.526106] ? do_raw_spin_unlock+0x75/0x190 [ 5139.526107] ? _raw_spin_unlock+0x12/0x30 [ 5139.526108] do_sys_openat2+0x2b4/0x410 [ 5139.526109] do_sys_open+0x73/0xa0 [ 5139.526110] __x64_sys_openat+0x24/0x30 [ 5139.526111] do_syscall_64+0xc1/0x440 [ 5139.526115] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5139.526117] RIP: 0033:0x7fdc0afbf332 [ 5139.526118] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5139.526121] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5139.526123] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5139.526125] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5139.526126] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5139.526131] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5139.526136] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5139.526146] NMI backtrace for cpu 0 [ 5139.526153] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5139.526160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5139.526162] Call Trace: [ 5139.526166] [ 5139.526169] ? dump_stack+0xbb/0x10e [ 5139.526176] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5139.526182] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5139.526188] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5139.526191] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5139.526196] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5139.526199] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5139.526205] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5139.526210] ? update_process_times+0x6d/0xc0 [ 5139.526217] ? tick_sched_handle.isra.18+0x27/0x90 [ 5139.526225] ? tick_sched_timer+0x53/0xb0 [ 5139.526230] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5139.526233] ? tick_sched_handle.isra.18+0x90/0x90 [ 5139.526238] ? hrtimer_interrupt+0x128/0x2c0 [ 5139.526243] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5139.526249] ? apic_timer_interrupt+0xf/0x20 [ 5139.526254] [ 5139.526257] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5139.526263] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5139.526267] ? dump_stack+0xf6/0x10e [ 5139.526271] ? do_raw_spin_lock+0x177/0x190 [ 5139.526275] ? _raw_spin_lock+0x21/0x30 [ 5139.526280] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5140.021444] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5140.024380] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5140.026881] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5140.030868] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5140.038751] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5140.041027] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5140.046829] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5140.050967] ? do_wait_intr+0xf0/0xf0 [ 5140.053826] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5140.060990] ? kthread+0x1d1/0x200 [ 5140.063787] ? set_kthread_struct+0x70/0x70 [ 5140.066600] ? ret_from_fork+0x1f/0x30 [ 5140.447618] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3115s! [ 5140.454747] Showing busy workqueues and worker pools: [ 5140.464557] workqueue events: flags=0x0 [ 5140.468244] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5140.470914] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5140.474017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5140.476691] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5140.480727] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5140.483902] pending: kfree_rcu_monitor [ 5140.485491] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5140.490375] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5140.495043] workqueue events_long: flags=0x0 [ 5140.496599] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5140.498674] pending: lu_objects_flush [obdclass] [ 5140.500887] workqueue events_unbound: flags=0x2 [ 5140.502728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5140.505107] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5140.509419] workqueue events_power_efficient: flags=0x80 [ 5140.511703] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5140.514317] pending: neigh_periodic_work, check_lifetime [ 5140.516733] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5140.519827] pending: do_cache_clean, neigh_periodic_work [ 5140.522408] workqueue mm_percpu_wq: flags=0x8 [ 5140.524819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5140.528224] pending: lru_add_drain_per_cpu [ 5140.530211] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5140.533309] pending: vmstat_update, lru_add_drain_per_cpu [ 5140.537927] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5140.541803] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5140.545186] workqueue writeback: flags=0x4a [ 5140.547394] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5140.551385] pending: wb_workfn, wb_workfn [ 5140.553938] workqueue xprtiod: flags=0xa [ 5140.559066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5140.565824] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5140.572272] workqueue ipv6_addrconf: flags=0x40008 [ 5140.575670] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5140.579957] pending: addrconf_verify_work [ 5140.582173] workqueue knbd0-recv: flags=0x1a [ 5140.588607] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5140.592198] in-flight: 655:recv_work [ 5140.594685] workqueue ptlrpc_pinger: flags=0x2 [ 5140.599896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5140.602661] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5140.607059] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3115s workers=4 idle: 36396 38 11188 [ 5140.613373] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5131s workers=2 idle: 658 [ 5152.053045] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 5152.056119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5152.104572] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5152.113976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5152.120971] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5152.124908] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5152.131318] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5152.148923] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5152.157406] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5152.165078] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5152.172043] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5152.178273] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5152.182599] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5152.187462] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5152.206066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5152.214323] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5152.222444] Call Trace: [ 5152.224665] [ 5152.226990] ? show_regs.cold.9+0x22/0x2f [ 5152.230721] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5152.233740] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5152.240070] ? watchdog+0x40/0x40 [ 5152.247058] ? hrtimer_interrupt+0x128/0x2c0 [ 5152.250805] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5152.256932] ? apic_timer_interrupt+0xf/0x20 [ 5152.259186] [ 5152.260223] ? native_safe_halt+0x2e/0x50 [ 5152.267759] kvm_wait+0xcd/0xe0 [ 5152.271012] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5152.275079] do_raw_spin_lock+0x177/0x190 [ 5152.279252] _raw_spin_lock+0x21/0x30 [ 5152.285322] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5152.298288] process_one_work+0x2c8/0x700 [ 5152.303429] worker_thread+0x296/0x6e0 [ 5152.308014] ? rescuer_thread+0x570/0x570 [ 5152.312211] kthread+0x1d1/0x200 [ 5152.314040] ? set_kthread_struct+0x70/0x70 [ 5152.319388] ret_from_fork+0x1f/0x30 [ 5156.054128] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5156.057119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5156.108809] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5156.120032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5156.140704] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5156.147789] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5156.161934] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5156.168606] RAX: 0000000000003610 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5156.178241] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5156.185531] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5156.192609] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5156.205375] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5156.209806] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5156.224758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5156.235922] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5156.238331] Call Trace: [ 5156.247371] [ 5156.248173] ? show_regs.cold.9+0x22/0x2f [ 5156.254023] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5156.258783] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5156.268876] ? watchdog+0x40/0x40 [ 5156.275289] ? hrtimer_interrupt+0x128/0x2c0 [ 5156.281664] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5156.283332] ? apic_timer_interrupt+0xf/0x20 [ 5156.293407] [ 5156.295704] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5156.299586] do_raw_spin_lock+0x177/0x190 [ 5156.301045] _raw_spin_lock+0x21/0x30 [ 5156.311676] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5156.321605] ? do_raw_spin_unlock+0x75/0x190 [ 5156.323092] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5156.341974] ? do_wait_intr+0xf0/0xf0 [ 5156.347841] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5156.353732] mdc_close+0x53b/0xd70 [mdc] [ 5156.358109] lmv_close+0x18d/0x490 [lmv] [ 5156.360137] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5156.372854] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5156.384779] ll_md_close+0x274/0x800 [lustre] [ 5156.386560] ll_file_release+0x128/0xc50 [lustre] [ 5156.405060] __fput+0x10d/0x440 [ 5156.411982] ____fput+0x12/0x20 [ 5156.417340] task_work_run+0xc6/0x110 [ 5156.422315] exit_to_usermode_loop+0x1dd/0x1f0 [ 5156.428437] do_syscall_64+0x3d6/0x440 [ 5156.429813] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5156.439974] RIP: 0033:0x7fdc0afbfcd8 [ 5156.448656] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5156.464114] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5156.477677] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5156.487802] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5156.491105] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5156.498706] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5156.501891] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5164.052209] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5164.056785] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5164.100014] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5164.114107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5164.123039] RIP: 0010:load_balance+0x725/0x1090 [ 5164.128518] Code: 48 83 00 01 9c 58 0f 1f 44 00 00 48 83 05 7a 6d dd 02 01 48 83 05 82 b9 dd 02 01 f6 c4 02 0f 85 22 03 00 00 48 8b 3c 24 57 9d <0f> 1f 44 00 00 8b 84 24 98 00 00 00 a8 02 74 23 83 e0 fd 48 83 05 [ 5164.145210] RSP: 0018:ffffba8e40d58de0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5164.149955] RAX: 0000000000000002 RBX: ffff9fe6141c2870 RCX: ffff9fe642133440 [ 5164.153987] RDX: 0000000000000008 RSI: ffff9fe6141c27c0 RDI: 0000000000000206 [ 5164.157790] RBP: ffffba8e40d58e90 R08: ffff9fe60131cf80 R09: 0000000000000000 [ 5164.164715] R10: ffff9fe6141c2858 R11: 000000000000a7fa R12: ffff9fe6141c27c0 [ 5164.169488] R13: ffff9fe6421b3e08 R14: 0000000000000008 R15: ffff9fe6421b3380 [ 5164.175069] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5164.180015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5164.183843] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5164.189360] Call Trace: [ 5164.190628] [ 5164.192580] ? show_regs.cold.9+0x22/0x2f [ 5164.195283] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5164.201040] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5164.203288] ? watchdog+0x40/0x40 [ 5164.205484] ? hrtimer_interrupt+0x128/0x2c0 [ 5164.207975] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5164.212610] ? apic_timer_interrupt+0xf/0x20 [ 5164.215490] ? apic_timer_interrupt+0xa/0x20 [ 5164.217736] ? load_balance+0x725/0x1090 [ 5164.220276] rebalance_domains+0x348/0x580 [ 5164.222503] ? kvm_clock_get_cycles+0x2c/0x50 [ 5164.227327] run_rebalance_domains+0x68/0xa0 [ 5164.235340] __do_softirq+0x134/0x4e8 [ 5164.237178] irq_exit_rcu+0x1c6/0x1e0 [ 5164.239961] irq_exit+0x12/0x30 [ 5164.241291] smp_apic_timer_interrupt+0xe0/0x270 [ 5164.246217] apic_timer_interrupt+0xf/0x20 [ 5164.250365] [ 5164.251510] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5164.254532] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5164.268298] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5164.275613] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5164.283550] RDX: 000000000000793c RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5164.291018] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5164.301130] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5164.305790] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5164.313006] do_raw_spin_lock+0x177/0x190 [ 5164.316341] _raw_spin_lock+0x21/0x30 [ 5164.318572] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5164.322128] ? do_raw_spin_unlock+0x75/0x190 [ 5164.327647] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5164.332490] ? do_wait_intr+0xf0/0xf0 [ 5164.334978] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5164.343341] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5164.349594] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5164.354384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5164.357440] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5164.359271] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5164.364350] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5164.367292] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5164.374035] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5164.384495] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5164.390927] lmv_intent_open+0x338/0x1150 [lmv] [ 5164.393392] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5164.398509] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5164.404872] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5164.410627] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5164.413442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5164.420197] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5164.422915] ? do_raw_spin_unlock+0x75/0x190 [ 5164.428289] ? _raw_spin_unlock+0x12/0x30 [ 5164.431798] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5164.435191] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5164.437777] ll_atomic_open+0x36a/0xd70 [lustre] [ 5164.444243] lookup_open+0x356/0xb40 [ 5164.446367] do_last+0x313/0xfc0 [ 5164.448734] ? nd_jump_root+0xe5/0x160 [ 5164.454509] ? path_init+0x437/0x520 [ 5164.457289] path_openat+0xf7/0x500 [ 5164.459766] ? do_raw_spin_unlock+0x75/0x190 [ 5164.463872] do_filp_open+0x99/0x140 [ 5164.465411] ? getname_flags+0x6e/0x330 [ 5164.470342] ? __check_object_size+0xff/0x256 [ 5164.472419] ? do_raw_spin_unlock+0x75/0x190 [ 5164.476497] ? _raw_spin_unlock+0x12/0x30 [ 5164.478571] do_sys_openat2+0x2b4/0x410 [ 5164.480731] do_sys_open+0x73/0xa0 [ 5164.485158] __x64_sys_openat+0x24/0x30 [ 5164.490404] do_syscall_64+0xc1/0x440 [ 5164.492695] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5164.497369] RIP: 0033:0x7fdc0afbf332 [ 5164.500199] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5164.515477] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5164.524764] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5164.532291] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5164.540321] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5164.549530] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5164.554650] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5168.049430] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 5168.054927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5168.082695] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5168.090764] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5168.096745] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5168.102833] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5168.116278] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5168.119063] RAX: 00000000000067a5 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5168.121781] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5168.124812] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5168.135449] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5168.138654] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5168.142501] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5168.145551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5168.147539] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5168.150725] Call Trace: [ 5168.151321] [ 5168.151817] ? show_regs.cold.9+0x22/0x2f [ 5168.152785] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5168.156816] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5168.158859] ? watchdog+0x40/0x40 [ 5168.161949] ? hrtimer_interrupt+0x128/0x2c0 [ 5168.167142] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5168.171393] ? apic_timer_interrupt+0xf/0x20 [ 5168.175703] [ 5168.181600] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5168.189860] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5168.202940] ? dump_stack+0xf6/0x10e [ 5168.207015] do_raw_spin_lock+0x177/0x190 [ 5168.214715] _raw_spin_lock+0x21/0x30 [ 5168.218217] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5168.228430] obd_mod_free+0x46/0x140 [ptlrpc] [ 5168.255411] mdc_commit_open+0x91/0xb0 [mdc] [ 5168.257037] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5168.281639] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5168.298411] after_reply+0xe2c/0x1780 [ptlrpc] [ 5168.311860] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5168.321662] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5168.332818] ? do_wait_intr+0xf0/0xf0 [ 5168.344683] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5168.360804] kthread+0x1d1/0x200 [ 5168.362005] ? set_kthread_struct+0x70/0x70 [ 5168.363474] ret_from_fork+0x1f/0x30 [ 5170.655394] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3145s! [ 5170.659122] Showing busy workqueues and worker pools: [ 5170.662015] workqueue events: flags=0x0 [ 5170.664992] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5170.667997] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5170.671527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5170.675668] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5170.681600] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5170.685481] pending: kfree_rcu_monitor [ 5170.687552] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5170.691670] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5170.699808] workqueue events_long: flags=0x0 [ 5170.702561] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5170.706161] pending: lu_objects_flush [obdclass] [ 5170.708627] workqueue events_unbound: flags=0x2 [ 5170.711106] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5170.714285] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5170.720404] workqueue events_power_efficient: flags=0x80 [ 5170.723545] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5170.726823] pending: neigh_periodic_work, check_lifetime [ 5170.729600] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5170.733202] pending: do_cache_clean, neigh_periodic_work [ 5170.736087] workqueue mm_percpu_wq: flags=0x8 [ 5170.738423] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5170.741899] pending: lru_add_drain_per_cpu [ 5170.744507] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5170.747585] pending: vmstat_update, lru_add_drain_per_cpu [ 5170.750505] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5170.753787] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5170.757103] workqueue writeback: flags=0x4a [ 5170.759207] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5170.764431] pending: wb_workfn, wb_workfn [ 5170.766754] workqueue xprtiod: flags=0xa [ 5170.769093] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5170.772311] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5170.776512] workqueue ipv6_addrconf: flags=0x40008 [ 5170.778578] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5170.781207] pending: addrconf_verify_work [ 5170.783123] workqueue knbd0-recv: flags=0x1a [ 5170.785177] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5170.788204] in-flight: 655:recv_work [ 5170.790194] workqueue ptlrpc_pinger: flags=0x2 [ 5170.795011] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5170.803166] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5170.805263] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3145s workers=4 idle: 36396 38 11188 [ 5170.823546] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5161s workers=2 idle: 658 [ 5180.051011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 5180.054826] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5180.104485] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5180.116262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5180.141426] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5180.155335] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5180.161029] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5180.174614] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5180.183613] RAX: 0000000000004d10 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5180.193203] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5180.205459] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5180.213521] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5180.224370] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5180.231423] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5180.237413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5180.247852] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5180.250524] Call Trace: [ 5180.257451] [ 5180.258735] ? show_regs.cold.9+0x22/0x2f [ 5180.265341] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5180.278961] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5180.285280] ? watchdog+0x40/0x40 [ 5180.289780] ? hrtimer_interrupt+0x128/0x2c0 [ 5180.293330] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5180.299423] ? apic_timer_interrupt+0xf/0x20 [ 5180.309461] [ 5180.313494] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5180.318476] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5180.322471] do_raw_spin_lock+0x177/0x190 [ 5180.324668] _raw_spin_lock+0x21/0x30 [ 5180.325872] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5180.330166] process_one_work+0x2c8/0x700 [ 5180.333440] worker_thread+0x296/0x6e0 [ 5180.339308] ? rescuer_thread+0x570/0x570 [ 5180.352268] kthread+0x1d1/0x200 [ 5180.358445] ? set_kthread_struct+0x70/0x70 [ 5180.367241] ret_from_fork+0x1f/0x30 [ 5184.054691] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5184.057399] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5184.085922] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5184.103983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5184.119937] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5184.125992] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5184.144005] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5184.151292] RAX: 0000000000001d83 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5184.163984] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5184.173966] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5184.180597] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5184.186387] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5184.193276] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5184.198043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5184.199580] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5184.207215] Call Trace: [ 5184.214445] [ 5184.216759] ? show_regs.cold.9+0x22/0x2f [ 5184.219397] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5184.230175] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5184.241007] ? watchdog+0x40/0x40 [ 5184.243401] ? hrtimer_interrupt+0x128/0x2c0 [ 5184.244783] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5184.256143] ? apic_timer_interrupt+0xf/0x20 [ 5184.258982] [ 5184.265789] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5184.268115] do_raw_spin_lock+0x177/0x190 [ 5184.275618] _raw_spin_lock+0x21/0x30 [ 5184.278181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5184.281394] ? do_raw_spin_unlock+0x75/0x190 [ 5184.284286] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5184.289028] ? do_wait_intr+0xf0/0xf0 [ 5184.291289] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5184.293396] mdc_close+0x53b/0xd70 [mdc] [ 5184.307932] lmv_close+0x18d/0x490 [lmv] [ 5184.313499] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5184.320418] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5184.327096] ll_md_close+0x274/0x800 [lustre] [ 5184.335415] ll_file_release+0x128/0xc50 [lustre] [ 5184.344656] __fput+0x10d/0x440 [ 5184.349413] ____fput+0x12/0x20 [ 5184.351490] task_work_run+0xc6/0x110 [ 5184.354038] exit_to_usermode_loop+0x1dd/0x1f0 [ 5184.360192] do_syscall_64+0x3d6/0x440 [ 5184.366775] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5184.374235] RIP: 0033:0x7fdc0afbfcd8 [ 5184.376243] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5184.397638] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5184.401107] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5184.410134] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5184.417291] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5184.425466] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5184.431175] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5192.052020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5192.055512] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5192.097684] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5192.110486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5192.116059] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5192.118738] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5192.136070] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5192.151933] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5192.162506] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5192.172288] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5192.180400] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5192.190468] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5192.199505] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5192.207903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5192.213725] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5192.231431] Call Trace: [ 5192.232858] [ 5192.235647] ? show_regs.cold.9+0x22/0x2f [ 5192.237096] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5192.243355] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5192.249425] ? watchdog+0x40/0x40 [ 5192.250693] ? hrtimer_interrupt+0x128/0x2c0 [ 5192.255802] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5192.257472] ? apic_timer_interrupt+0xf/0x20 [ 5192.259808] [ 5192.261388] ? native_safe_halt+0x2e/0x50 [ 5192.265182] kvm_wait+0xcd/0xe0 [ 5192.269276] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5192.271216] do_raw_spin_lock+0x177/0x190 [ 5192.273187] _raw_spin_lock+0x21/0x30 [ 5192.275720] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5192.281959] ? do_raw_spin_unlock+0x75/0x190 [ 5192.283438] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5192.286192] ? do_wait_intr+0xf0/0xf0 [ 5192.287508] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5192.293115] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5192.298139] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5192.303424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5192.306188] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5192.309001] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5192.311145] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5192.313815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5192.315905] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5192.319037] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5192.321694] lmv_intent_open+0x338/0x1150 [lmv] [ 5192.323816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5192.328171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5192.334080] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5192.340681] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5192.343655] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5192.349219] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5192.351855] ? do_raw_spin_unlock+0x75/0x190 [ 5192.354994] ? _raw_spin_unlock+0x12/0x30 [ 5192.357919] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5192.362091] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5192.364795] ll_atomic_open+0x36a/0xd70 [lustre] [ 5192.367802] lookup_open+0x356/0xb40 [ 5192.370002] do_last+0x313/0xfc0 [ 5192.371766] ? nd_jump_root+0xe5/0x160 [ 5192.374547] ? path_init+0x437/0x520 [ 5192.376599] path_openat+0xf7/0x500 [ 5192.378474] ? do_raw_spin_unlock+0x75/0x190 [ 5192.382601] do_filp_open+0x99/0x140 [ 5192.386135] ? getname_flags+0x6e/0x330 [ 5192.391177] ? __check_object_size+0xff/0x256 [ 5192.394035] ? do_raw_spin_unlock+0x75/0x190 [ 5192.397794] ? _raw_spin_unlock+0x12/0x30 [ 5192.402643] do_sys_openat2+0x2b4/0x410 [ 5192.404900] do_sys_open+0x73/0xa0 [ 5192.406724] __x64_sys_openat+0x24/0x30 [ 5192.408780] do_syscall_64+0xc1/0x440 [ 5192.411972] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5192.415402] RIP: 0033:0x7fdc0afbf332 [ 5192.419600] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5192.432476] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5192.437728] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5192.442284] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5192.446188] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5192.450844] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5192.455429] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5196.052246] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 5196.069196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5196.113373] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5196.125305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5196.131273] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5196.137858] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5196.158653] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5196.161690] RAX: 0000000000007312 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5196.168305] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5196.171725] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5196.175853] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5196.183996] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5196.189609] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5196.192848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5196.198450] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5196.204905] Call Trace: [ 5196.208093] [ 5196.211214] ? show_regs.cold.9+0x22/0x2f [ 5196.213003] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5196.219246] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5196.220896] ? watchdog+0x40/0x40 [ 5196.224395] ? hrtimer_interrupt+0x128/0x2c0 [ 5196.226378] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5196.232285] ? apic_timer_interrupt+0xf/0x20 [ 5196.235236] [ 5196.236055] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5196.246571] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5196.248920] ? dump_stack+0xf6/0x10e [ 5196.250393] do_raw_spin_lock+0x177/0x190 [ 5196.255887] _raw_spin_lock+0x21/0x30 [ 5196.257075] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5196.258847] obd_mod_free+0x46/0x140 [ptlrpc] [ 5196.268274] mdc_commit_open+0x91/0xb0 [mdc] [ 5196.270640] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5196.272457] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5196.274957] after_reply+0xe2c/0x1780 [ptlrpc] [ 5196.277048] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5196.283144] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5196.286051] ? do_wait_intr+0xf0/0xf0 [ 5196.287515] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5196.289769] kthread+0x1d1/0x200 [ 5196.291142] ? set_kthread_struct+0x70/0x70 [ 5196.292989] ret_from_fork+0x1f/0x30 [ 5200.865059] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3175s! [ 5200.871496] Showing busy workqueues and worker pools: [ 5200.876237] workqueue events: flags=0x0 [ 5200.881518] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5200.884389] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5200.890799] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5200.897153] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5200.904494] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5200.910857] pending: kfree_rcu_monitor [ 5200.915938] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5200.922634] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5200.931701] workqueue events_long: flags=0x0 [ 5200.937016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5200.943601] pending: lu_objects_flush [obdclass] [ 5200.946097] workqueue events_unbound: flags=0x2 [ 5200.951613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5200.957727] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5200.966036] workqueue events_power_efficient: flags=0x80 [ 5200.971922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5200.974329] pending: neigh_periodic_work, check_lifetime [ 5200.980286] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5200.986630] pending: do_cache_clean, neigh_periodic_work [ 5200.992579] workqueue mm_percpu_wq: flags=0x8 [ 5200.994642] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5201.001175] pending: lru_add_drain_per_cpu [ 5201.009974] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5201.012620] pending: vmstat_update, lru_add_drain_per_cpu [ 5201.017850] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5201.023633] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5201.026071] workqueue writeback: flags=0x4a [ 5201.031407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5201.037577] pending: wb_workfn, wb_workfn [ 5201.039708] workqueue xprtiod: flags=0xa [ 5201.043566] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5201.050544] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5201.056706] workqueue ipv6_addrconf: flags=0x40008 [ 5201.059703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5201.065870] pending: addrconf_verify_work [ 5201.070645] workqueue knbd0-recv: flags=0x1a [ 5201.072754] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5201.078541] in-flight: 655:recv_work [ 5201.079947] workqueue ptlrpc_pinger: flags=0x2 [ 5201.085329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5201.087448] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5201.092810] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3176s workers=4 idle: 36396 38 11188 [ 5201.095740] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5191s workers=2 idle: 658 [ 5208.057094] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5208.067197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5208.124171] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5208.142569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5208.152363] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5208.163071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5208.168133] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5208.192140] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5208.211116] RAX: 0000000000001992 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5208.226372] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5208.233524] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5208.244202] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5208.262142] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5208.274153] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5208.288649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5208.298025] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5208.302586] Call Trace: [ 5208.307752] [ 5208.309574] ? show_regs.cold.9+0x22/0x2f [ 5208.315195] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5208.316916] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5208.329134] ? watchdog+0x40/0x40 [ 5208.330264] ? hrtimer_interrupt+0x128/0x2c0 [ 5208.336925] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5208.342778] ? apic_timer_interrupt+0xf/0x20 [ 5208.349328] [ 5208.353648] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5208.359943] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5208.361626] do_raw_spin_lock+0x177/0x190 [ 5208.374309] _raw_spin_lock+0x21/0x30 [ 5208.378149] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5208.386914] process_one_work+0x2c8/0x700 [ 5208.389894] worker_thread+0x296/0x6e0 [ 5208.393544] ? rescuer_thread+0x570/0x570 [ 5208.396508] kthread+0x1d1/0x200 [ 5208.407170] ? set_kthread_struct+0x70/0x70 [ 5208.408694] ret_from_fork+0x1f/0x30 [ 5212.054033] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5212.061030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5212.101456] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5212.109919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5212.115398] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5212.116909] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5212.125006] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5212.127947] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5212.130752] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5212.133444] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5212.138370] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5212.141466] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5212.147653] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5212.151547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5212.160842] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5212.165969] Call Trace: [ 5212.174456] [ 5212.180805] ? show_regs.cold.9+0x22/0x2f [ 5212.185314] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5212.195645] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5212.197378] ? watchdog+0x40/0x40 [ 5212.210755] ? hrtimer_interrupt+0x128/0x2c0 [ 5212.215542] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5212.226267] ? apic_timer_interrupt+0xf/0x20 [ 5212.227638] [ 5212.228397] ? native_safe_halt+0x2e/0x50 [ 5212.243706] kvm_wait+0xcd/0xe0 [ 5212.247389] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5212.254745] do_raw_spin_lock+0x177/0x190 [ 5212.256115] _raw_spin_lock+0x21/0x30 [ 5212.275118] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5212.285253] ? do_raw_spin_unlock+0x75/0x190 [ 5212.286761] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5212.292625] ? do_wait_intr+0xf0/0xf0 [ 5212.295661] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5212.306690] mdc_close+0x53b/0xd70 [mdc] [ 5212.315139] lmv_close+0x18d/0x490 [lmv] [ 5212.319682] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5212.332256] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5212.340872] ll_md_close+0x274/0x800 [lustre] [ 5212.342661] ll_file_release+0x128/0xc50 [lustre] [ 5212.345300] __fput+0x10d/0x440 [ 5212.346987] ____fput+0x12/0x20 [ 5212.351313] task_work_run+0xc6/0x110 [ 5212.356411] exit_to_usermode_loop+0x1dd/0x1f0 [ 5212.363549] do_syscall_64+0x3d6/0x440 [ 5212.368077] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5212.369796] RIP: 0033:0x7fdc0afbfcd8 [ 5212.371429] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5212.390145] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5212.398972] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5212.403903] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5212.408764] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5212.414965] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5212.428597] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5220.056128] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5220.065113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5220.118130] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5220.134300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5220.150527] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5220.152590] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5220.176769] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5220.186352] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5220.196616] RDX: 00000000000052e0 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5220.202822] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5220.212382] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5220.217614] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5220.226578] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5220.238384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5220.241707] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5220.246266] Call Trace: [ 5220.247792] [ 5220.248724] ? show_regs.cold.9+0x22/0x2f [ 5220.250170] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5220.252774] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5220.253995] ? watchdog+0x40/0x40 [ 5220.254843] ? hrtimer_interrupt+0x128/0x2c0 [ 5220.258549] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5220.259626] ? apic_timer_interrupt+0xf/0x20 [ 5220.260673] [ 5220.261782] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5220.263780] do_raw_spin_lock+0x177/0x190 [ 5220.271359] _raw_spin_lock+0x21/0x30 [ 5220.272706] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5220.282297] ? do_raw_spin_unlock+0x75/0x190 [ 5220.287371] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5220.290117] ? do_wait_intr+0xf0/0xf0 [ 5220.291189] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5220.297559] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5220.300812] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5220.310164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5220.312134] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5220.320272] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5220.324030] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5220.334956] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5220.340014] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5220.346477] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5220.349821] lmv_intent_open+0x338/0x1150 [lmv] [ 5220.350939] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5220.352412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5220.364513] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5220.373716] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5220.381749] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5220.387000] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5220.395984] ? do_raw_spin_unlock+0x75/0x190 [ 5220.406705] ? _raw_spin_unlock+0x12/0x30 [ 5220.409997] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5220.412416] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5220.414577] ll_atomic_open+0x36a/0xd70 [lustre] [ 5220.416322] lookup_open+0x356/0xb40 [ 5220.422185] do_last+0x313/0xfc0 [ 5220.423325] ? nd_jump_root+0xe5/0x160 [ 5220.432186] ? path_init+0x437/0x520 [ 5220.437529] path_openat+0xf7/0x500 [ 5220.441313] ? do_raw_spin_unlock+0x75/0x190 [ 5220.447730] do_filp_open+0x99/0x140 [ 5220.451255] ? getname_flags+0x6e/0x330 [ 5220.453951] ? __check_object_size+0xff/0x256 [ 5220.461844] ? do_raw_spin_unlock+0x75/0x190 [ 5220.464461] ? _raw_spin_unlock+0x12/0x30 [ 5220.469289] do_sys_openat2+0x2b4/0x410 [ 5220.472875] do_sys_open+0x73/0xa0 [ 5220.482507] __x64_sys_openat+0x24/0x30 [ 5220.486745] do_syscall_64+0xc1/0x440 [ 5220.491157] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5220.495704] RIP: 0033:0x7fdc0afbf332 [ 5220.502208] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5220.521066] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5220.526670] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5220.536238] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5220.547799] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5220.554697] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5220.562108] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5228.049291] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 5228.052629] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5228.099552] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5228.104048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5228.106793] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5228.108661] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5228.116576] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5228.119858] RAX: 0000000000006207 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5228.122889] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5228.125623] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5228.131266] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5228.134204] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5228.137237] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5228.141080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5228.143661] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5228.147139] Call Trace: [ 5228.148218] [ 5228.149082] ? show_regs.cold.9+0x22/0x2f [ 5228.150718] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5228.152704] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5228.154595] ? watchdog+0x40/0x40 [ 5228.156057] ? hrtimer_interrupt+0x128/0x2c0 [ 5228.157623] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5228.159544] ? apic_timer_interrupt+0xf/0x20 [ 5228.161120] [ 5228.161862] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5228.163963] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5228.166016] ? dump_stack+0xf6/0x10e [ 5228.167338] do_raw_spin_lock+0x177/0x190 [ 5228.168788] _raw_spin_lock+0x21/0x30 [ 5228.170221] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5228.172208] obd_mod_free+0x46/0x140 [ptlrpc] [ 5228.174549] mdc_commit_open+0x91/0xb0 [mdc] [ 5228.177872] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5228.182466] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5228.189584] after_reply+0xe2c/0x1780 [ptlrpc] [ 5228.194734] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5228.199684] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5228.204786] ? do_wait_intr+0xf0/0xf0 [ 5228.208017] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5228.213354] kthread+0x1d1/0x200 [ 5228.215609] ? set_kthread_struct+0x70/0x70 [ 5228.220618] ret_from_fork+0x1f/0x30 [ 5231.584039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3206s! [ 5231.589948] Showing busy workqueues and worker pools: [ 5231.592826] workqueue events: flags=0x0 [ 5231.595594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5231.598713] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5231.601571] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5231.604528] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5231.609103] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5231.612035] pending: kfree_rcu_monitor [ 5231.613934] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5231.616810] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5231.622415] workqueue events_long: flags=0x0 [ 5231.624370] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5231.627640] pending: lu_objects_flush [obdclass] [ 5231.630479] workqueue events_unbound: flags=0x2 [ 5231.632727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5231.635551] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5231.640446] workqueue events_power_efficient: flags=0x80 [ 5231.644263] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5231.648200] pending: neigh_periodic_work, check_lifetime [ 5231.650834] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5231.653775] pending: do_cache_clean, neigh_periodic_work [ 5231.656448] workqueue mm_percpu_wq: flags=0x8 [ 5231.658608] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5231.661805] pending: lru_add_drain_per_cpu [ 5231.664102] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5231.667419] pending: vmstat_update, lru_add_drain_per_cpu [ 5231.670691] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5231.674425] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5231.678180] workqueue writeback: flags=0x4a [ 5231.680610] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5231.684242] pending: wb_workfn, wb_workfn [ 5231.686855] workqueue xprtiod: flags=0xa [ 5231.689907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5231.693513] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5231.697190] workqueue ipv6_addrconf: flags=0x40008 [ 5231.699517] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5231.702743] pending: addrconf_verify_work [ 5231.704767] workqueue knbd0-recv: flags=0x1a [ 5231.706671] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5231.709553] in-flight: 655:recv_work [ 5231.711414] workqueue ptlrpc_pinger: flags=0x2 [ 5231.713481] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5231.716192] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5231.718782] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3206s workers=4 idle: 36396 38 11188 [ 5231.722192] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5222s workers=2 idle: 658 [ 5236.057460] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5236.067650] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5236.130616] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5236.153432] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5236.166421] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5236.172490] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5236.190704] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5236.224139] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5236.238674] RAX: 0000000000007da9 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5236.256205] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5236.271374] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5236.283658] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5236.285806] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5236.301696] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5236.311801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5236.322875] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5236.334468] Call Trace: [ 5236.339999] [ 5236.340710] ? show_regs.cold.9+0x22/0x2f [ 5236.346007] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5236.353379] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5236.359091] ? watchdog+0x40/0x40 [ 5236.362762] ? hrtimer_interrupt+0x128/0x2c0 [ 5236.368620] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5236.374276] ? apic_timer_interrupt+0xf/0x20 [ 5236.379237] [ 5236.381329] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5236.388749] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5236.396415] do_raw_spin_lock+0x177/0x190 [ 5236.402835] _raw_spin_lock+0x21/0x30 [ 5236.408146] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5236.416240] process_one_work+0x2c8/0x700 [ 5236.421430] worker_thread+0x296/0x6e0 [ 5236.428146] ? rescuer_thread+0x570/0x570 [ 5236.431466] kthread+0x1d1/0x200 [ 5236.436263] ? set_kthread_struct+0x70/0x70 [ 5236.438449] ret_from_fork+0x1f/0x30 [ 5240.054007] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5240.059446] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5240.103918] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5240.111919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5240.118890] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5240.125862] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5240.143201] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5240.153572] RAX: 0000000000007904 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5240.163244] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5240.167017] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5240.174180] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5240.185580] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5240.190286] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5240.201071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5240.204349] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5240.209430] Call Trace: [ 5240.211323] [ 5240.212355] ? show_regs.cold.9+0x22/0x2f [ 5240.217178] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5240.220070] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5240.222535] ? watchdog+0x40/0x40 [ 5240.225063] ? hrtimer_interrupt+0x128/0x2c0 [ 5240.231762] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5240.235142] ? apic_timer_interrupt+0xf/0x20 [ 5240.240045] [ 5240.241558] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5240.245584] do_raw_spin_lock+0x177/0x190 [ 5240.251126] _raw_spin_lock+0x21/0x30 [ 5240.253203] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5240.257160] ? do_raw_spin_unlock+0x75/0x190 [ 5240.273845] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5240.275893] ? do_wait_intr+0xf0/0xf0 [ 5240.283902] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5240.287389] mdc_close+0x53b/0xd70 [mdc] [ 5240.289913] lmv_close+0x18d/0x490 [lmv] [ 5240.300189] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5240.313522] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5240.317518] ll_md_close+0x274/0x800 [lustre] [ 5240.320634] ll_file_release+0x128/0xc50 [lustre] [ 5240.322236] __fput+0x10d/0x440 [ 5240.326660] ____fput+0x12/0x20 [ 5240.330453] task_work_run+0xc6/0x110 [ 5240.334271] exit_to_usermode_loop+0x1dd/0x1f0 [ 5240.336352] do_syscall_64+0x3d6/0x440 [ 5240.337497] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5240.339294] RIP: 0033:0x7fdc0afbfcd8 [ 5240.340870] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5240.355357] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5240.366758] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5240.369307] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5240.379723] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5240.388373] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5240.394561] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5248.052038] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5248.055000] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5248.088827] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5248.099088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5248.104438] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5248.108725] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5248.119628] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5248.127733] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5248.133386] RDX: 0000000000000dcb RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5248.138411] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5248.142368] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5248.146376] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5248.149709] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5248.153553] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5248.156781] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5248.159766] Call Trace: [ 5248.161271] [ 5248.165090] ? show_regs.cold.9+0x22/0x2f [ 5248.166702] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5248.173562] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5248.175098] ? watchdog+0x40/0x40 [ 5248.183772] ? hrtimer_interrupt+0x128/0x2c0 [ 5248.185352] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5248.187411] ? apic_timer_interrupt+0xf/0x20 [ 5248.189157] [ 5248.189942] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5248.192752] do_raw_spin_lock+0x177/0x190 [ 5248.195034] _raw_spin_lock+0x21/0x30 [ 5248.196636] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5248.199439] ? do_raw_spin_unlock+0x75/0x190 [ 5248.201347] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5248.204010] ? do_wait_intr+0xf0/0xf0 [ 5248.208329] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5248.214751] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5248.222931] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5248.227500] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5248.233013] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5248.235914] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5248.238316] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5248.240384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5248.243681] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5248.246723] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5248.249557] lmv_intent_open+0x338/0x1150 [lmv] [ 5248.251497] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5248.254207] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5248.258033] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5248.260170] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5248.262371] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5248.264745] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5248.270412] ? do_raw_spin_unlock+0x75/0x190 [ 5248.272912] ? _raw_spin_unlock+0x12/0x30 [ 5248.275720] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5248.279072] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5248.281453] ll_atomic_open+0x36a/0xd70 [lustre] [ 5248.284269] lookup_open+0x356/0xb40 [ 5248.286209] do_last+0x313/0xfc0 [ 5248.288253] ? nd_jump_root+0xe5/0x160 [ 5248.290288] ? path_init+0x437/0x520 [ 5248.292218] path_openat+0xf7/0x500 [ 5248.293341] ? do_raw_spin_unlock+0x75/0x190 [ 5248.295420] do_filp_open+0x99/0x140 [ 5248.297302] ? getname_flags+0x6e/0x330 [ 5248.299426] ? __check_object_size+0xff/0x256 [ 5248.302012] ? do_raw_spin_unlock+0x75/0x190 [ 5248.304102] ? _raw_spin_unlock+0x12/0x30 [ 5248.305716] do_sys_openat2+0x2b4/0x410 [ 5248.307905] do_sys_open+0x73/0xa0 [ 5248.309489] __x64_sys_openat+0x24/0x30 [ 5248.311302] do_syscall_64+0xc1/0x440 [ 5248.312653] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5248.315170] RIP: 0033:0x7fdc0afbf332 [ 5248.316732] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5248.327049] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5248.330707] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5248.333858] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5248.336979] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5248.340014] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5248.343050] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5256.048340] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 5256.060992] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5256.095706] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5256.101933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5256.106891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5256.111893] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5256.126324] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5256.136817] RAX: 0000000000006584 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5256.141871] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5256.154142] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5256.167869] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5256.178832] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5256.186798] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5256.195345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5256.204874] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5256.220622] Call Trace: [ 5256.221597] [ 5256.224301] ? show_regs.cold.9+0x22/0x2f [ 5256.226859] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5256.233346] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5256.243127] ? watchdog+0x40/0x40 [ 5256.249446] ? hrtimer_interrupt+0x128/0x2c0 [ 5256.254244] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5256.263250] ? apic_timer_interrupt+0xf/0x20 [ 5256.269389] [ 5256.270158] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5256.278340] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5256.285460] ? dump_stack+0xf6/0x10e [ 5256.294213] do_raw_spin_lock+0x177/0x190 [ 5256.298368] _raw_spin_lock+0x21/0x30 [ 5256.302890] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5256.310255] obd_mod_free+0x46/0x140 [ptlrpc] [ 5256.312174] mdc_commit_open+0x91/0xb0 [mdc] [ 5256.316254] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5256.321710] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5256.334321] after_reply+0xe2c/0x1780 [ptlrpc] [ 5256.342020] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5256.352696] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5256.356431] ? do_wait_intr+0xf0/0xf0 [ 5256.357329] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5256.358796] kthread+0x1d1/0x200 [ 5256.372373] ? set_kthread_struct+0x70/0x70 [ 5256.378381] ret_from_fork+0x1f/0x30 [ 5261.791691] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3236s! [ 5261.807613] Showing busy workqueues and worker pools: [ 5261.809533] workqueue events: flags=0x0 [ 5261.818512] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5261.820961] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5261.832965] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5261.840442] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5261.851029] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5261.860104] pending: kfree_rcu_monitor [ 5261.861572] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5261.870594] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5261.900643] workqueue events_long: flags=0x0 [ 5261.902164] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5261.910618] pending: lu_objects_flush [obdclass] [ 5261.918934] workqueue events_unbound: flags=0x2 [ 5261.923985] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5261.933980] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5261.953806] workqueue events_power_efficient: flags=0x80 [ 5261.962242] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5261.982921] pending: neigh_periodic_work, check_lifetime [ 5261.984971] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5262.002533] pending: do_cache_clean, neigh_periodic_work [ 5262.010116] workqueue mm_percpu_wq: flags=0x8 [ 5262.011689] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5262.020115] pending: lru_add_drain_per_cpu [ 5262.031599] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5262.036156] pending: vmstat_update, lru_add_drain_per_cpu [ 5262.043811] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5262.052432] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5262.065410] workqueue writeback: flags=0x4a [ 5262.066902] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5262.077214] pending: wb_workfn, wb_workfn [ 5262.078670] workqueue xprtiod: flags=0xa [ 5262.083687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5262.093584] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5262.101860] workqueue ipv6_addrconf: flags=0x40008 [ 5262.104818] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5262.114644] pending: addrconf_verify_work [ 5262.116067] workqueue knbd0-recv: flags=0x1a [ 5262.126264] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5262.132299] in-flight: 655:recv_work [ 5262.144303] workqueue ptlrpc_pinger: flags=0x2 [ 5262.151661] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5262.153785] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5262.172297] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3237s workers=4 idle: 36396 38 11188 [ 5262.187187] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5252s workers=2 idle: 658 [ 5264.056394] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5264.062491] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5264.107722] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5264.118228] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5264.130453] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5264.139392] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5264.152071] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5264.175699] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5264.187184] RAX: 0000000000001b0b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5264.196170] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5264.206729] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5264.217632] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5264.223822] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5264.228353] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5264.243782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5264.245743] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5264.249822] Call Trace: [ 5264.252859] [ 5264.253943] ? show_regs.cold.9+0x22/0x2f [ 5264.255429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5264.257519] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5264.267101] ? watchdog+0x40/0x40 [ 5264.269910] ? hrtimer_interrupt+0x128/0x2c0 [ 5264.272795] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5264.283407] ? apic_timer_interrupt+0xf/0x20 [ 5264.284874] [ 5264.285611] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5264.289483] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5264.295659] do_raw_spin_lock+0x177/0x190 [ 5264.302221] _raw_spin_lock+0x21/0x30 [ 5264.304220] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5264.307594] process_one_work+0x2c8/0x700 [ 5264.313390] worker_thread+0x296/0x6e0 [ 5264.317310] ? rescuer_thread+0x570/0x570 [ 5264.318827] kthread+0x1d1/0x200 [ 5264.323822] ? set_kthread_struct+0x70/0x70 [ 5264.331750] ret_from_fork+0x1f/0x30 [ 5268.054550] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5268.067544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5268.110740] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5268.116449] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5268.121048] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5268.124709] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5268.134582] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5268.137980] RAX: 0000000000003546 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5268.141870] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5268.158129] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5268.164347] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5268.168760] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5268.180724] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5268.189178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5268.198436] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5268.203522] Call Trace: [ 5268.210955] [ 5268.212221] ? show_regs.cold.9+0x22/0x2f [ 5268.223798] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5268.235872] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5268.247946] ? watchdog+0x40/0x40 [ 5268.256762] ? hrtimer_interrupt+0x128/0x2c0 [ 5268.263441] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5268.271950] ? apic_timer_interrupt+0xf/0x20 [ 5268.280915] [ 5268.287777] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5268.289745] do_raw_spin_lock+0x177/0x190 [ 5268.306230] _raw_spin_lock+0x21/0x30 [ 5268.307524] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5268.322288] ? do_raw_spin_unlock+0x75/0x190 [ 5268.325394] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5268.330751] ? do_wait_intr+0xf0/0xf0 [ 5268.339573] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5268.348828] mdc_close+0x53b/0xd70 [mdc] [ 5268.351958] lmv_close+0x18d/0x490 [lmv] [ 5268.359508] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5268.364851] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5268.373721] ll_md_close+0x274/0x800 [lustre] [ 5268.383919] ll_file_release+0x128/0xc50 [lustre] [ 5268.390719] __fput+0x10d/0x440 [ 5268.393397] ____fput+0x12/0x20 [ 5268.394968] task_work_run+0xc6/0x110 [ 5268.401108] exit_to_usermode_loop+0x1dd/0x1f0 [ 5268.407858] do_syscall_64+0x3d6/0x440 [ 5268.409781] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5268.415250] RIP: 0033:0x7fdc0afbfcd8 [ 5268.419897] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5268.442414] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5268.449897] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5268.457384] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5268.468168] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5268.480962] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5268.492138] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5276.056080] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5276.061447] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5276.101755] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5276.106528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5276.110554] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5276.113448] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5276.132876] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5276.136454] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5276.138718] RDX: 0000000000004b1e RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5276.141691] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5276.145691] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5276.152403] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5276.158350] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5276.169748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5276.180428] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5276.185582] Call Trace: [ 5276.187040] [ 5276.188355] ? show_regs.cold.9+0x22/0x2f [ 5276.194771] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5276.198414] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5276.200503] ? watchdog+0x40/0x40 [ 5276.202688] ? hrtimer_interrupt+0x128/0x2c0 [ 5276.213645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5276.221547] ? apic_timer_interrupt+0xf/0x20 [ 5276.226505] [ 5276.227725] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5276.242927] do_raw_spin_lock+0x177/0x190 [ 5276.245717] _raw_spin_lock+0x21/0x30 [ 5276.247936] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5276.250862] ? do_raw_spin_unlock+0x75/0x190 [ 5276.252231] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5276.253937] ? do_wait_intr+0xf0/0xf0 [ 5276.255753] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5276.257879] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5276.259978] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5276.262770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5276.264899] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5276.267334] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5276.272652] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5276.276908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5276.282734] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5276.288762] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5276.292016] lmv_intent_open+0x338/0x1150 [lmv] [ 5276.293972] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5276.296713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5276.299481] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5276.301426] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5276.307008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5276.311185] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5276.318216] ? do_raw_spin_unlock+0x75/0x190 [ 5276.320672] ? _raw_spin_unlock+0x12/0x30 [ 5276.324714] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5276.326906] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5276.328991] ll_atomic_open+0x36a/0xd70 [lustre] [ 5276.335765] lookup_open+0x356/0xb40 [ 5276.337355] do_last+0x313/0xfc0 [ 5276.338601] ? nd_jump_root+0xe5/0x160 [ 5276.340552] ? path_init+0x437/0x520 [ 5276.342240] path_openat+0xf7/0x500 [ 5276.343599] ? do_raw_spin_unlock+0x75/0x190 [ 5276.345010] do_filp_open+0x99/0x140 [ 5276.346465] ? getname_flags+0x6e/0x330 [ 5276.347807] ? __check_object_size+0xff/0x256 [ 5276.349575] ? do_raw_spin_unlock+0x75/0x190 [ 5276.351298] ? _raw_spin_unlock+0x12/0x30 [ 5276.352649] do_sys_openat2+0x2b4/0x410 [ 5276.356103] do_sys_open+0x73/0xa0 [ 5276.357137] __x64_sys_openat+0x24/0x30 [ 5276.365503] do_syscall_64+0xc1/0x440 [ 5276.366616] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5276.368772] RIP: 0033:0x7fdc0afbf332 [ 5276.373779] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5276.385535] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5276.389711] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5276.397448] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5276.404660] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5276.412050] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5276.417390] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5288.048031] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5288.052086] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5288.081436] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5288.087437] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5288.092613] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5288.096012] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5288.104243] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5288.106571] RAX: 0000000000005789 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5288.109725] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5288.112625] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5288.115807] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5288.118418] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5288.121332] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5288.124463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5288.126832] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5288.129567] Call Trace: [ 5288.130502] [ 5288.131276] ? show_regs.cold.9+0x22/0x2f [ 5288.132721] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5288.135365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5288.137412] ? watchdog+0x40/0x40 [ 5288.139019] ? hrtimer_interrupt+0x128/0x2c0 [ 5288.140553] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5288.141818] ? apic_timer_interrupt+0xf/0x20 [ 5288.143576] [ 5288.145508] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5288.148375] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5288.150994] ? dump_stack+0xf6/0x10e [ 5288.152579] do_raw_spin_lock+0x177/0x190 [ 5288.154336] _raw_spin_lock+0x21/0x30 [ 5288.156215] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5288.158830] obd_mod_free+0x46/0x140 [ptlrpc] [ 5288.161204] mdc_commit_open+0x91/0xb0 [mdc] [ 5288.163272] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5288.165780] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5288.168327] after_reply+0xe2c/0x1780 [ptlrpc] [ 5288.170696] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5288.173352] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5288.175401] ? do_wait_intr+0xf0/0xf0 [ 5288.176812] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5288.179377] kthread+0x1d1/0x200 [ 5288.180667] ? set_kthread_struct+0x70/0x70 [ 5288.182186] ret_from_fork+0x1f/0x30 [ 5292.052045] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5292.060974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5292.094863] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5292.099650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5292.102996] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5292.105768] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5292.108991] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5292.116604] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5292.119982] RAX: 0000000000003932 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5292.122688] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5292.125510] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5292.128376] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5292.131640] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5292.134695] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5292.138247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5292.140465] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5292.143509] Call Trace: [ 5292.144526] [ 5292.145407] ? show_regs.cold.9+0x22/0x2f [ 5292.147015] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5292.149375] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5292.151431] ? watchdog+0x40/0x40 [ 5292.152980] ? hrtimer_interrupt+0x128/0x2c0 [ 5292.154471] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5292.156508] ? apic_timer_interrupt+0xf/0x20 [ 5292.158591] [ 5292.159691] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5292.161994] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5292.164480] do_raw_spin_lock+0x177/0x190 [ 5292.166163] _raw_spin_lock+0x21/0x30 [ 5292.167700] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5292.170520] process_one_work+0x2c8/0x700 [ 5292.172248] worker_thread+0x296/0x6e0 [ 5292.173907] ? rescuer_thread+0x570/0x570 [ 5292.175710] kthread+0x1d1/0x200 [ 5292.177241] ? set_kthread_struct+0x70/0x70 [ 5292.178768] ret_from_fork+0x1f/0x30 [ 5292.515157] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3267s! [ 5292.529049] Showing busy workqueues and worker pools: [ 5292.533769] workqueue events: flags=0x0 [ 5292.541360] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5292.550233] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5292.559984] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5292.573096] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5292.593147] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5292.602012] pending: kfree_rcu_monitor [ 5292.608623] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5292.621007] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5292.635387] workqueue events_long: flags=0x0 [ 5292.638610] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5292.646489] pending: lu_objects_flush [obdclass] [ 5292.655530] workqueue events_unbound: flags=0x2 [ 5292.657083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5292.665600] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5292.680028] workqueue events_power_efficient: flags=0x80 [ 5292.685597] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5292.695126] pending: neigh_periodic_work, check_lifetime [ 5292.696974] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5292.707656] pending: do_cache_clean, neigh_periodic_work [ 5292.714966] workqueue mm_percpu_wq: flags=0x8 [ 5292.716238] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5292.729740] pending: lru_add_drain_per_cpu [ 5292.736381] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5292.743354] pending: vmstat_update, lru_add_drain_per_cpu [ 5292.746082] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5292.749211] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5292.753269] workqueue writeback: flags=0x4a [ 5292.756571] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5292.760418] pending: wb_workfn, wb_workfn [ 5292.767518] workqueue xprtiod: flags=0xa [ 5292.778615] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5292.780761] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5292.786582] workqueue ipv6_addrconf: flags=0x40008 [ 5292.788510] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5292.791262] pending: addrconf_verify_work [ 5292.793638] workqueue knbd0-recv: flags=0x1a [ 5292.797076] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5292.803680] in-flight: 655:recv_work [ 5292.808695] workqueue ptlrpc_pinger: flags=0x2 [ 5292.810543] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5292.817701] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5292.825083] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3267s workers=4 idle: 36396 38 11188 [ 5292.837428] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5283s workers=2 idle: 658 [ 5296.058457] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5296.072984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5296.130368] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5296.148331] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5296.163030] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5296.167516] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5296.183513] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5296.190866] RAX: 000000000000220a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5296.194909] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5296.207172] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5296.216181] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5296.221272] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5296.224049] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5296.233391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5296.239191] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5296.250847] Call Trace: [ 5296.252357] [ 5296.253232] ? show_regs.cold.9+0x22/0x2f [ 5296.261286] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5296.264595] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5296.269940] ? watchdog+0x40/0x40 [ 5296.274112] ? hrtimer_interrupt+0x128/0x2c0 [ 5296.284331] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5296.290681] ? apic_timer_interrupt+0xf/0x20 [ 5296.300087] [ 5296.301891] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5296.305078] do_raw_spin_lock+0x177/0x190 [ 5296.308396] _raw_spin_lock+0x21/0x30 [ 5296.314707] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5296.320248] ? do_raw_spin_unlock+0x75/0x190 [ 5296.324041] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5296.327164] ? do_wait_intr+0xf0/0xf0 [ 5296.329023] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5296.332818] mdc_close+0x53b/0xd70 [mdc] [ 5296.336255] lmv_close+0x18d/0x490 [lmv] [ 5296.341698] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5296.349056] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5296.355414] ll_md_close+0x274/0x800 [lustre] [ 5296.357943] ll_file_release+0x128/0xc50 [lustre] [ 5296.366366] __fput+0x10d/0x440 [ 5296.367251] ____fput+0x12/0x20 [ 5296.368037] task_work_run+0xc6/0x110 [ 5296.372681] exit_to_usermode_loop+0x1dd/0x1f0 [ 5296.375474] do_syscall_64+0x3d6/0x440 [ 5296.378042] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5296.380514] RIP: 0033:0x7fdc0afbfcd8 [ 5296.382618] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5296.396248] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5296.400036] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5296.407267] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5296.412202] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5296.418506] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5296.430196] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5304.052099] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5304.063276] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5304.140501] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5304.146689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5304.164669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5304.170369] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5304.206588] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5304.217787] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5304.229755] RDX: 0000000000001ee4 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5304.231920] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5304.242470] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5304.245348] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5304.258070] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5304.266164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5304.270405] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5304.282937] Call Trace: [ 5304.284614] [ 5304.285378] ? show_regs.cold.9+0x22/0x2f [ 5304.287186] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5304.291322] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5304.295065] ? watchdog+0x40/0x40 [ 5304.296380] ? hrtimer_interrupt+0x128/0x2c0 [ 5304.298287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5304.304556] ? apic_timer_interrupt+0xf/0x20 [ 5304.307432] [ 5304.308169] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5304.309929] do_raw_spin_lock+0x177/0x190 [ 5304.312375] _raw_spin_lock+0x21/0x30 [ 5304.313663] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5304.316731] ? do_raw_spin_unlock+0x75/0x190 [ 5304.321566] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5304.328135] ? do_wait_intr+0xf0/0xf0 [ 5304.332966] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5304.335619] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5304.346049] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5304.351789] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5304.361036] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5304.367618] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5304.371147] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5304.376142] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5304.384038] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5304.393660] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5304.398561] lmv_intent_open+0x338/0x1150 [lmv] [ 5304.404693] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5304.408172] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5304.412840] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5304.417972] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5304.421753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5304.425500] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5304.430083] ? do_raw_spin_unlock+0x75/0x190 [ 5304.432029] ? _raw_spin_unlock+0x12/0x30 [ 5304.433725] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5304.436499] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5304.441343] ll_atomic_open+0x36a/0xd70 [lustre] [ 5304.446237] lookup_open+0x356/0xb40 [ 5304.447484] do_last+0x313/0xfc0 [ 5304.451186] ? nd_jump_root+0xe5/0x160 [ 5304.454224] ? path_init+0x437/0x520 [ 5304.459179] path_openat+0xf7/0x500 [ 5304.460479] ? do_raw_spin_unlock+0x75/0x190 [ 5304.461890] do_filp_open+0x99/0x140 [ 5304.464462] ? getname_flags+0x6e/0x330 [ 5304.467238] ? __check_object_size+0xff/0x256 [ 5304.469144] ? do_raw_spin_unlock+0x75/0x190 [ 5304.470829] ? _raw_spin_unlock+0x12/0x30 [ 5304.472385] do_sys_openat2+0x2b4/0x410 [ 5304.476458] do_sys_open+0x73/0xa0 [ 5304.477815] __x64_sys_openat+0x24/0x30 [ 5304.481051] do_syscall_64+0xc1/0x440 [ 5304.487681] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5304.491428] RIP: 0033:0x7fdc0afbf332 [ 5304.496386] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5304.516141] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5304.531613] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5304.535462] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5304.540564] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5304.548900] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5304.552691] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5319.421315] rcu: INFO: rcu_sched self-detected stall on CPU [ 5319.424195] rcu: 0-...!: (1959628 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 5319.431696] (t=3300078 jiffies g=270761 q=3486) [ 5319.436045] rcu: rcu_sched kthread starved for 3295010 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 5319.446224] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5319.458428] rcu: RCU grace-period kthread stack dump: [ 5319.462778] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5319.468640] Call Trace: [ 5319.470435] __schedule+0x351/0xcb0 [ 5319.472219] schedule+0xc0/0x180 [ 5319.474093] schedule_timeout+0xb4/0x190 [ 5319.475600] ? do_raw_spin_unlock+0x75/0x190 [ 5319.478254] ? __next_timer_interrupt+0x160/0x160 [ 5319.480262] ? prepare_to_swait_event+0xf3/0x140 [ 5319.482407] rcu_gp_kthread+0x8dd/0x15b0 [ 5319.484234] ? force_qs_rnp+0x2c0/0x2c0 [ 5319.485975] kthread+0x1d1/0x200 [ 5319.487575] ? set_kthread_struct+0x70/0x70 [ 5319.489421] ret_from_fork+0x1f/0x30 [ 5319.491022] rcu: Stack dump where RCU GP kthread last ran: [ 5319.494218] Sending NMI from CPU 0 to CPUs 2: [ 5319.496303] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 5319.497235] NMI backtrace for cpu 0 [ 5319.503114] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5319.507967] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5319.511895] Call Trace: [ 5319.513077] [ 5319.514211] ? dump_stack+0xbb/0x10e [ 5319.516135] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5319.518884] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5319.521011] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5319.523655] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5319.526050] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5319.528810] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5319.530994] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5319.533271] ? update_process_times+0x6d/0xc0 [ 5319.535216] ? tick_sched_handle.isra.18+0x27/0x90 [ 5319.539871] ? tick_sched_timer+0x53/0xb0 [ 5319.541827] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5319.544217] ? tick_sched_handle.isra.18+0x90/0x90 [ 5319.546286] ? hrtimer_interrupt+0x128/0x2c0 [ 5319.548568] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5319.550706] ? apic_timer_interrupt+0xf/0x20 [ 5319.552567] [ 5319.553601] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5319.555938] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5319.558451] ? dump_stack+0xf6/0x10e [ 5319.560044] ? do_raw_spin_lock+0x177/0x190 [ 5319.561803] ? _raw_spin_lock+0x21/0x30 [ 5319.563526] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5319.566940] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5319.569105] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5319.571982] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5319.574687] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5319.577331] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5319.579855] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5319.582207] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5319.584884] ? do_wait_intr+0xf0/0xf0 [ 5319.587288] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5319.591348] ? kthread+0x1d1/0x200 [ 5319.593428] ? set_kthread_struct+0x70/0x70 [ 5319.595594] ? ret_from_fork+0x1f/0x30 [ 5320.051066] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5320.056612] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5320.096585] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5320.101683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5320.106961] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5320.111208] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5320.113910] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5320.137842] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5320.149911] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5320.158619] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5320.164587] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5320.172802] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5320.182850] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5320.188669] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5320.198683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5320.206830] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5320.217670] Call Trace: [ 5320.222779] [ 5320.226244] ? show_regs.cold.9+0x22/0x2f [ 5320.230866] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5320.240012] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5320.243229] ? watchdog+0x40/0x40 [ 5320.244561] ? hrtimer_interrupt+0x128/0x2c0 [ 5320.246663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5320.248904] ? apic_timer_interrupt+0xf/0x20 [ 5320.250855] [ 5320.251863] ? native_safe_halt+0x2e/0x50 [ 5320.253761] kvm_wait+0xcd/0xe0 [ 5320.256334] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5320.258742] do_raw_spin_lock+0x177/0x190 [ 5320.260578] _raw_spin_lock+0x21/0x30 [ 5320.261641] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5320.263837] process_one_work+0x2c8/0x700 [ 5320.265525] worker_thread+0x296/0x6e0 [ 5320.270942] ? rescuer_thread+0x570/0x570 [ 5320.272777] kthread+0x1d1/0x200 [ 5320.277853] ? set_kthread_struct+0x70/0x70 [ 5320.281471] ret_from_fork+0x1f/0x30 [ 5323.238054] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3298s! [ 5323.250496] Showing busy workqueues and worker pools: [ 5323.252300] workqueue events: flags=0x0 [ 5323.260537] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5323.267125] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5323.279970] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5323.285092] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5323.300844] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5323.305605] pending: kfree_rcu_monitor [ 5323.307057] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5323.314180] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5323.329542] workqueue events_long: flags=0x0 [ 5323.331463] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5323.340043] pending: lu_objects_flush [obdclass] [ 5323.350769] workqueue events_unbound: flags=0x2 [ 5323.354939] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5323.369250] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5323.378463] workqueue events_power_efficient: flags=0x80 [ 5323.380854] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5323.399005] pending: neigh_periodic_work, check_lifetime [ 5323.401038] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5323.407544] pending: do_cache_clean, neigh_periodic_work [ 5323.416138] workqueue mm_percpu_wq: flags=0x8 [ 5323.423621] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5323.436064] pending: lru_add_drain_per_cpu [ 5323.442438] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5323.456937] pending: vmstat_update, lru_add_drain_per_cpu [ 5323.458864] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5323.473041] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5323.483673] workqueue writeback: flags=0x4a [ 5323.485018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5323.497202] pending: wb_workfn, wb_workfn [ 5323.498529] workqueue xprtiod: flags=0xa [ 5323.502932] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5323.508907] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5323.524409] workqueue ipv6_addrconf: flags=0x40008 [ 5323.526081] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5323.537016] pending: addrconf_verify_work [ 5323.546581] workqueue knbd0-recv: flags=0x1a [ 5323.550597] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5323.556352] in-flight: 655:recv_work [ 5323.557678] workqueue ptlrpc_pinger: flags=0x2 [ 5323.571803] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5323.588278] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5323.596593] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3298s workers=4 idle: 36396 38 11188 [ 5323.602541] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5314s workers=2 idle: 658 [ 5324.054022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5324.056837] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5324.097701] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5324.111220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5324.120413] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5324.134084] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5324.155835] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5324.170436] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5324.183448] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5324.201850] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5324.209354] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5324.216458] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5324.230145] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5324.242766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5324.251919] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5324.270252] Call Trace: [ 5324.271288] [ 5324.276190] ? show_regs.cold.9+0x22/0x2f [ 5324.282990] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5324.303130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5324.308292] ? watchdog+0x40/0x40 [ 5324.312815] ? hrtimer_interrupt+0x128/0x2c0 [ 5324.321342] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5324.328376] ? apic_timer_interrupt+0xf/0x20 [ 5324.333853] [ 5324.336216] ? native_safe_halt+0x2e/0x50 [ 5324.341271] kvm_wait+0xcd/0xe0 [ 5324.346868] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5324.350545] do_raw_spin_lock+0x177/0x190 [ 5324.355415] _raw_spin_lock+0x21/0x30 [ 5324.356464] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5324.360215] ? do_raw_spin_unlock+0x75/0x190 [ 5324.364936] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5324.372327] ? do_wait_intr+0xf0/0xf0 [ 5324.373611] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5324.384553] mdc_close+0x53b/0xd70 [mdc] [ 5324.387386] lmv_close+0x18d/0x490 [lmv] [ 5324.389040] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5324.394972] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5324.398875] ll_md_close+0x274/0x800 [lustre] [ 5324.401383] ll_file_release+0x128/0xc50 [lustre] [ 5324.405688] __fput+0x10d/0x440 [ 5324.410173] ____fput+0x12/0x20 [ 5324.413663] task_work_run+0xc6/0x110 [ 5324.415017] exit_to_usermode_loop+0x1dd/0x1f0 [ 5324.416445] do_syscall_64+0x3d6/0x440 [ 5324.420165] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5324.425005] RIP: 0033:0x7fdc0afbfcd8 [ 5324.430694] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5324.449464] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5324.454528] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5324.459038] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5324.463787] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5324.470606] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5324.480056] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5344.052020] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 5344.054465] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5344.068029] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5344.073750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5344.077456] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5344.080021] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5344.090513] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5344.096007] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5344.107537] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5344.113394] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5344.117033] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5344.125538] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5344.133616] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5344.143553] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5344.152247] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5344.157428] Call Trace: [ 5344.158243] [ 5344.162703] ? show_regs.cold.9+0x22/0x2f [ 5344.165118] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5344.171400] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5344.179337] ? watchdog+0x40/0x40 [ 5344.182610] ? hrtimer_interrupt+0x128/0x2c0 [ 5344.186879] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5344.189456] ? apic_timer_interrupt+0xf/0x20 [ 5344.193609] [ 5344.195228] ? native_safe_halt+0x2e/0x50 [ 5344.196637] kvm_wait+0xcd/0xe0 [ 5344.198952] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5344.201648] do_raw_spin_lock+0x177/0x190 [ 5344.203521] _raw_spin_lock+0x21/0x30 [ 5344.205467] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5344.210557] ? do_raw_spin_unlock+0x75/0x190 [ 5344.214002] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5344.217941] ? do_wait_intr+0xf0/0xf0 [ 5344.219584] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5344.223236] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5344.225376] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5344.228509] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5344.231954] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5344.234549] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5344.236993] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5344.239303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5344.242804] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5344.249716] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5344.253380] lmv_intent_open+0x338/0x1150 [lmv] [ 5344.258271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5344.268523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5344.278206] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5344.280223] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5344.282071] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5344.283989] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5344.286232] ? do_raw_spin_unlock+0x75/0x190 [ 5344.288633] ? _raw_spin_unlock+0x12/0x30 [ 5344.290564] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5344.292994] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5344.295118] ll_atomic_open+0x36a/0xd70 [lustre] [ 5344.298518] lookup_open+0x356/0xb40 [ 5344.300384] do_last+0x313/0xfc0 [ 5344.303384] ? nd_jump_root+0xe5/0x160 [ 5344.307027] ? path_init+0x437/0x520 [ 5344.310580] path_openat+0xf7/0x500 [ 5344.313149] ? do_raw_spin_unlock+0x75/0x190 [ 5344.314841] do_filp_open+0x99/0x140 [ 5344.316422] ? getname_flags+0x6e/0x330 [ 5344.321430] ? __check_object_size+0xff/0x256 [ 5344.327024] ? do_raw_spin_unlock+0x75/0x190 [ 5344.329817] ? _raw_spin_unlock+0x12/0x30 [ 5344.331600] do_sys_openat2+0x2b4/0x410 [ 5344.334586] do_sys_open+0x73/0xa0 [ 5344.339295] __x64_sys_openat+0x24/0x30 [ 5344.341263] do_syscall_64+0xc1/0x440 [ 5344.344492] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5344.346737] RIP: 0033:0x7fdc0afbf332 [ 5344.347947] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5344.365580] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5344.376261] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5344.382427] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5344.392490] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5344.403405] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5344.411854] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5348.048016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5348.050667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5348.054307] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5348.091177] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5348.091190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5348.091194] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5348.091207] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5348.091396] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5348.091403] RAX: 0000000000001114 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5348.091405] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5348.091407] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5348.091409] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5348.091411] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5348.091414] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5348.091416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5348.091418] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5348.091426] Call Trace: [ 5348.091431] [ 5348.091437] ? show_regs.cold.9+0x22/0x2f [ 5348.091447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5348.091452] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5348.091456] ? watchdog+0x40/0x40 [ 5348.091460] ? hrtimer_interrupt+0x128/0x2c0 [ 5348.091464] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5348.091473] ? apic_timer_interrupt+0xf/0x20 [ 5348.091478] [ 5348.091481] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5348.091486] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5348.091496] ? dump_stack+0xf6/0x10e [ 5348.091502] do_raw_spin_lock+0x177/0x190 [ 5348.091506] _raw_spin_lock+0x21/0x30 [ 5348.091514] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5348.091734] obd_mod_free+0x46/0x140 [ptlrpc] [ 5348.092290] mdc_commit_open+0x91/0xb0 [mdc] [ 5348.092325] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5348.092493] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5348.092630] after_reply+0xe2c/0x1780 [ptlrpc] [ 5348.098381] Modules linked in: [ 5348.107543] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5348.119858] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5348.119934] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5348.119941] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5348.119948] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5348.120225] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5348.120236] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5348.120240] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5348.120245] RAX: 0000000000007f95 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5348.120247] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5348.120249] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5348.120251] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5348.120253] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5348.120256] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5348.120258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5348.120261] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5348.120269] Call Trace: [ 5348.120275] [ 5348.120280] ? show_regs.cold.9+0x22/0x2f [ 5348.120291] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5348.120298] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5348.120303] ? watchdog+0x40/0x40 [ 5348.120307] ? hrtimer_interrupt+0x128/0x2c0 [ 5348.120311] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5348.120320] ? apic_timer_interrupt+0xf/0x20 [ 5348.120324] [ 5348.120328] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5348.120333] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5348.120338] do_raw_spin_lock+0x177/0x190 [ 5348.120343] _raw_spin_lock+0x21/0x30 [ 5348.120348] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5348.124681] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5348.125157] ? do_wait_intr+0xf0/0xf0 [ 5348.125171] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5348.125434] kthread+0x1d1/0x200 [ 5348.125440] ? set_kthread_struct+0x70/0x70 [ 5348.125444] ret_from_fork+0x1f/0x30 [ 5348.552996] process_one_work+0x2c8/0x700 [ 5348.555968] worker_thread+0x296/0x6e0 [ 5348.557307] ? rescuer_thread+0x570/0x570 [ 5348.559890] kthread+0x1d1/0x200 [ 5348.561613] ? set_kthread_struct+0x70/0x70 [ 5348.564010] ret_from_fork+0x1f/0x30 [ 5352.054074] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5352.063630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5352.101620] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5352.114454] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5352.120589] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5352.125592] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5352.143683] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5352.146374] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5352.149700] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5352.153965] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5352.157578] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5352.165366] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5352.168113] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5352.172246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5352.177674] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5352.187720] Call Trace: [ 5352.188573] [ 5352.190714] ? show_regs.cold.9+0x22/0x2f [ 5352.193009] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5352.194356] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5352.199964] ? watchdog+0x40/0x40 [ 5352.201249] ? hrtimer_interrupt+0x128/0x2c0 [ 5352.205246] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5352.208234] ? apic_timer_interrupt+0xf/0x20 [ 5352.210333] [ 5352.211638] ? native_safe_halt+0x2e/0x50 [ 5352.212780] kvm_wait+0xcd/0xe0 [ 5352.214576] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5352.220842] do_raw_spin_lock+0x177/0x190 [ 5352.222715] _raw_spin_lock+0x21/0x30 [ 5352.224155] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5352.225789] ? do_raw_spin_unlock+0x75/0x190 [ 5352.227716] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5352.230360] ? do_wait_intr+0xf0/0xf0 [ 5352.231816] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5352.234241] mdc_close+0x53b/0xd70 [mdc] [ 5352.235730] lmv_close+0x18d/0x490 [lmv] [ 5352.237327] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5352.242510] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5352.246249] ll_md_close+0x274/0x800 [lustre] [ 5352.248198] ll_file_release+0x128/0xc50 [lustre] [ 5352.251556] __fput+0x10d/0x440 [ 5352.252673] ____fput+0x12/0x20 [ 5352.253864] task_work_run+0xc6/0x110 [ 5352.256846] exit_to_usermode_loop+0x1dd/0x1f0 [ 5352.260579] do_syscall_64+0x3d6/0x440 [ 5352.262709] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5352.265633] RIP: 0033:0x7fdc0afbfcd8 [ 5352.267960] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5352.276161] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5352.279334] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5352.282176] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5352.285333] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5352.288136] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5352.290981] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5353.951334] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3329s! [ 5353.958809] Showing busy workqueues and worker pools: [ 5353.961869] workqueue events: flags=0x0 [ 5353.963427] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5353.974092] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5353.979015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5353.986375] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5354.004238] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5354.011232] pending: kfree_rcu_monitor [ 5354.014543] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5354.020478] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5354.032266] workqueue events_long: flags=0x0 [ 5354.038009] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5354.042432] pending: lu_objects_flush [obdclass] [ 5354.048554] workqueue events_unbound: flags=0x2 [ 5354.051788] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5354.055773] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5354.063254] workqueue events_power_efficient: flags=0x80 [ 5354.071090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5354.079807] pending: neigh_periodic_work, check_lifetime [ 5354.081926] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5354.086089] pending: do_cache_clean, neigh_periodic_work [ 5354.089409] workqueue mm_percpu_wq: flags=0x8 [ 5354.094422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5354.102244] pending: lru_add_drain_per_cpu [ 5354.105053] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5354.108075] pending: vmstat_update, lru_add_drain_per_cpu [ 5354.115236] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5354.119848] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5354.136677] workqueue writeback: flags=0x4a [ 5354.142567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5354.148058] pending: wb_workfn, wb_workfn [ 5354.152507] workqueue xprtiod: flags=0xa [ 5354.153451] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5354.165970] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5354.177680] workqueue ipv6_addrconf: flags=0x40008 [ 5354.187925] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5354.197830] pending: addrconf_verify_work [ 5354.202665] workqueue knbd0-recv: flags=0x1a [ 5354.208438] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5354.213429] in-flight: 655:recv_work [ 5354.217149] workqueue ptlrpc_pinger: flags=0x2 [ 5354.224827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5354.234485] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5354.244599] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3329s workers=4 idle: 36396 38 11188 [ 5354.250551] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5344s workers=2 idle: 658 [ 5372.052069] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 5372.055828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5372.097574] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5372.109933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5372.115588] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5372.118045] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5372.141856] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5372.148554] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5372.154566] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5372.158537] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5372.162940] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5372.166143] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5372.172740] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5372.176568] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5372.183522] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5372.191769] Call Trace: [ 5372.194063] [ 5372.196846] ? show_regs.cold.9+0x22/0x2f [ 5372.199003] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5372.202033] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5372.204080] ? watchdog+0x40/0x40 [ 5372.205288] ? hrtimer_interrupt+0x128/0x2c0 [ 5372.208732] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5372.211332] ? apic_timer_interrupt+0xf/0x20 [ 5372.213581] [ 5372.214579] ? native_safe_halt+0x2e/0x50 [ 5372.216619] kvm_wait+0xcd/0xe0 [ 5372.217823] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5372.219969] do_raw_spin_lock+0x177/0x190 [ 5372.221602] _raw_spin_lock+0x21/0x30 [ 5372.222979] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5372.225445] ? do_raw_spin_unlock+0x75/0x190 [ 5372.227242] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5372.229188] ? do_wait_intr+0xf0/0xf0 [ 5372.230210] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5372.232353] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5372.234741] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5372.238041] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5372.241534] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5372.245580] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5372.247570] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5372.249236] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5372.253971] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5372.257977] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5372.262926] lmv_intent_open+0x338/0x1150 [lmv] [ 5372.264494] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5372.266181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5372.270757] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5372.274601] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5372.278873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5372.283167] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5372.285420] ? do_raw_spin_unlock+0x75/0x190 [ 5372.288088] ? _raw_spin_unlock+0x12/0x30 [ 5372.289553] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5372.293197] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5372.296178] ll_atomic_open+0x36a/0xd70 [lustre] [ 5372.300846] lookup_open+0x356/0xb40 [ 5372.301674] do_last+0x313/0xfc0 [ 5372.304846] ? nd_jump_root+0xe5/0x160 [ 5372.306600] ? path_init+0x437/0x520 [ 5372.307961] path_openat+0xf7/0x500 [ 5372.308888] ? do_raw_spin_unlock+0x75/0x190 [ 5372.310548] do_filp_open+0x99/0x140 [ 5372.312917] ? getname_flags+0x6e/0x330 [ 5372.314947] ? __check_object_size+0xff/0x256 [ 5372.318596] ? do_raw_spin_unlock+0x75/0x190 [ 5372.324826] ? _raw_spin_unlock+0x12/0x30 [ 5372.326043] do_sys_openat2+0x2b4/0x410 [ 5372.328251] do_sys_open+0x73/0xa0 [ 5372.330676] __x64_sys_openat+0x24/0x30 [ 5372.335853] do_syscall_64+0xc1/0x440 [ 5372.340533] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5372.341982] RIP: 0033:0x7fdc0afbf332 [ 5372.343086] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5372.349985] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5372.354074] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5372.362175] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5372.366276] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5372.372872] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5372.375750] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5376.051027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5376.053210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5376.084497] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5376.095959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5376.100572] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5376.107775] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5376.109286] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5376.117955] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5376.121080] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5376.123981] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5376.126953] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5376.129893] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5376.132914] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5376.135869] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5376.138662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5376.142509] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5376.145937] Call Trace: [ 5376.147052] [ 5376.147994] ? show_regs.cold.9+0x22/0x2f [ 5376.149417] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5376.151245] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5376.153019] ? watchdog+0x40/0x40 [ 5376.154299] ? hrtimer_interrupt+0x128/0x2c0 [ 5376.155850] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5376.158046] ? apic_timer_interrupt+0xf/0x20 [ 5376.159798] [ 5376.160624] ? native_safe_halt+0x2e/0x50 [ 5376.162136] kvm_wait+0xcd/0xe0 [ 5376.163491] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5376.168567] do_raw_spin_lock+0x177/0x190 [ 5376.170186] _raw_spin_lock+0x21/0x30 [ 5376.173930] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5376.176056] process_one_work+0x2c8/0x700 [ 5376.181856] worker_thread+0x296/0x6e0 [ 5376.186154] ? rescuer_thread+0x570/0x570 [ 5376.187811] kthread+0x1d1/0x200 [ 5376.189183] ? set_kthread_struct+0x70/0x70 [ 5376.190828] ret_from_fork+0x1f/0x30 [ 5380.048026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5380.052736] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5380.059908] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5380.090415] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5380.090428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5380.090431] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5380.090443] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5380.090447] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5380.090452] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5380.090454] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5380.090456] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5380.090458] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5380.090460] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5380.090463] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5380.090465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5380.090467] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5380.090474] Call Trace: [ 5380.090479] [ 5380.090484] ? show_regs.cold.9+0x22/0x2f [ 5380.090493] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5380.090498] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5380.090502] ? watchdog+0x40/0x40 [ 5380.090506] ? hrtimer_interrupt+0x128/0x2c0 [ 5380.090509] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5380.090515] ? apic_timer_interrupt+0xf/0x20 [ 5380.090518] [ 5380.090521] ? native_safe_halt+0x2e/0x50 [ 5380.090524] kvm_wait+0xcd/0xe0 [ 5380.090530] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5380.090536] ? dump_stack+0xf6/0x10e [ 5380.090540] do_raw_spin_lock+0x177/0x190 [ 5380.090544] _raw_spin_lock+0x21/0x30 [ 5380.090550] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5380.090821] obd_mod_free+0x46/0x140 [ptlrpc] [ 5380.091004] mdc_commit_open+0x91/0xb0 [mdc] [ 5380.091048] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5380.091239] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5380.091434] after_reply+0xe2c/0x1780 [ptlrpc] [ 5380.091629] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5380.091837] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5380.092014] ? do_wait_intr+0xf0/0xf0 [ 5380.092022] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5380.092175] kthread+0x1d1/0x200 [ 5380.092183] ? set_kthread_struct+0x70/0x70 [ 5380.092186] ret_from_fork+0x1f/0x30 [ 5380.296562] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5380.315334] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5380.319723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5380.323337] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5380.325844] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5380.333428] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5380.336762] RAX: 000000000000411c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5380.340091] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5380.343211] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5380.354699] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5380.365176] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5380.368580] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5380.373832] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5380.377702] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5380.382022] Call Trace: [ 5380.384789] [ 5380.386588] ? show_regs.cold.9+0x22/0x2f [ 5380.388996] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5380.394159] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5380.397637] ? watchdog+0x40/0x40 [ 5380.400338] ? hrtimer_interrupt+0x128/0x2c0 [ 5380.404701] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5380.406325] ? apic_timer_interrupt+0xf/0x20 [ 5380.408798] [ 5380.409707] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5380.412974] do_raw_spin_lock+0x177/0x190 [ 5380.415700] _raw_spin_lock+0x21/0x30 [ 5380.418662] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5380.423623] ? do_raw_spin_unlock+0x75/0x190 [ 5380.427043] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5380.431633] ? do_wait_intr+0xf0/0xf0 [ 5380.434669] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5380.438916] mdc_close+0x53b/0xd70 [mdc] [ 5380.445156] lmv_close+0x18d/0x490 [lmv] [ 5380.453622] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5380.455169] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5380.465315] ll_md_close+0x274/0x800 [lustre] [ 5380.467706] ll_file_release+0x128/0xc50 [lustre] [ 5380.469867] __fput+0x10d/0x440 [ 5380.475052] ____fput+0x12/0x20 [ 5380.476314] task_work_run+0xc6/0x110 [ 5380.479218] exit_to_usermode_loop+0x1dd/0x1f0 [ 5380.486067] do_syscall_64+0x3d6/0x440 [ 5380.487515] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5380.491397] RIP: 0033:0x7fdc0afbfcd8 [ 5380.493136] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5380.505243] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5380.509028] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5380.512635] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5380.515137] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5380.518575] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5380.521548] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5384.671021] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3359s! [ 5384.673945] Showing busy workqueues and worker pools: [ 5384.678819] workqueue events: flags=0x0 [ 5384.682710] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5384.690026] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5384.699178] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5384.706616] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5384.719138] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5384.726505] pending: kfree_rcu_monitor [ 5384.730184] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5384.735368] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5384.748050] workqueue events_long: flags=0x0 [ 5384.753770] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5384.761960] pending: lu_objects_flush [obdclass] [ 5384.765089] workqueue events_unbound: flags=0x2 [ 5384.767818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5384.771465] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5384.776468] workqueue events_power_efficient: flags=0x80 [ 5384.779090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5384.782075] pending: neigh_periodic_work, check_lifetime [ 5384.785098] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5384.788260] pending: do_cache_clean, neigh_periodic_work [ 5384.792069] workqueue mm_percpu_wq: flags=0x8 [ 5384.794113] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5384.797276] pending: lru_add_drain_per_cpu [ 5384.799453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5384.801812] pending: vmstat_update, lru_add_drain_per_cpu [ 5384.806931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5384.809681] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5384.812383] workqueue writeback: flags=0x4a [ 5384.814295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5384.817876] pending: wb_workfn, wb_workfn [ 5384.819988] workqueue xprtiod: flags=0xa [ 5384.823458] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5384.825323] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5384.827971] workqueue ipv6_addrconf: flags=0x40008 [ 5384.829547] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5384.842144] pending: addrconf_verify_work [ 5384.845035] workqueue knbd0-recv: flags=0x1a [ 5384.848442] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5384.852474] in-flight: 655:recv_work [ 5384.855052] workqueue ptlrpc_pinger: flags=0x2 [ 5384.859685] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5384.865611] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5384.869610] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3359s workers=4 idle: 36396 38 11188 [ 5384.874659] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5375s workers=2 idle: 658 [ 5400.052025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 5400.054756] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5400.072789] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5400.076851] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5400.079837] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5400.081479] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5400.088911] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5400.091626] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5400.094483] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5400.097395] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5400.100219] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5400.102782] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5400.105453] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5400.108710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5400.111084] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5400.113853] Call Trace: [ 5400.114986] [ 5400.115557] ? show_regs.cold.9+0x22/0x2f [ 5400.116635] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5400.117986] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5400.119467] ? watchdog+0x40/0x40 [ 5400.120706] ? hrtimer_interrupt+0x128/0x2c0 [ 5400.122426] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5400.124373] ? apic_timer_interrupt+0xf/0x20 [ 5400.126068] [ 5400.126794] ? native_safe_halt+0x2e/0x50 [ 5400.128390] kvm_wait+0xcd/0xe0 [ 5400.129600] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5400.131660] do_raw_spin_lock+0x177/0x190 [ 5400.133266] _raw_spin_lock+0x21/0x30 [ 5400.134965] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5400.137268] ? do_raw_spin_unlock+0x75/0x190 [ 5400.139197] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5400.141398] ? do_wait_intr+0xf0/0xf0 [ 5400.142848] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5400.145163] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5400.147341] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5400.149697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5400.151852] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5400.154184] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5400.155846] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5400.157479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5400.159610] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5400.161868] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5400.164229] lmv_intent_open+0x338/0x1150 [lmv] [ 5400.165977] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5400.168159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5400.170112] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5400.171759] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5400.173499] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5400.175562] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5400.177169] ? do_raw_spin_unlock+0x75/0x190 [ 5400.178728] ? _raw_spin_unlock+0x12/0x30 [ 5400.179842] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5400.181191] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5400.182609] ll_atomic_open+0x36a/0xd70 [lustre] [ 5400.184041] lookup_open+0x356/0xb40 [ 5400.185310] do_last+0x313/0xfc0 [ 5400.186289] ? nd_jump_root+0xe5/0x160 [ 5400.187452] ? path_init+0x437/0x520 [ 5400.188462] path_openat+0xf7/0x500 [ 5400.189560] ? do_raw_spin_unlock+0x75/0x190 [ 5400.190849] do_filp_open+0x99/0x140 [ 5400.192216] ? getname_flags+0x6e/0x330 [ 5400.193593] ? __check_object_size+0xff/0x256 [ 5400.195217] ? do_raw_spin_unlock+0x75/0x190 [ 5400.196953] ? _raw_spin_unlock+0x12/0x30 [ 5400.198580] do_sys_openat2+0x2b4/0x410 [ 5400.200055] do_sys_open+0x73/0xa0 [ 5400.201263] __x64_sys_openat+0x24/0x30 [ 5400.202536] do_syscall_64+0xc1/0x440 [ 5400.203959] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5400.205838] RIP: 0033:0x7fdc0afbf332 [ 5400.207346] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5400.214067] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5400.217130] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5400.219997] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5400.222460] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5400.224629] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5400.227422] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5404.051014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5404.072926] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5404.099059] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5404.104015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5404.112018] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5404.116508] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5404.120126] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5404.133820] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5404.140546] RAX: 0000000000003947 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5404.144158] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5404.148638] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5404.153279] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5404.157573] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5404.162774] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5404.167108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5404.170069] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5404.173523] Call Trace: [ 5404.174745] [ 5404.175852] ? show_regs.cold.9+0x22/0x2f [ 5404.177703] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5404.179870] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5404.181858] ? watchdog+0x40/0x40 [ 5404.182877] ? hrtimer_interrupt+0x128/0x2c0 [ 5404.184858] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5404.187255] ? apic_timer_interrupt+0xf/0x20 [ 5404.191127] [ 5404.192218] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5404.195131] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5404.197683] do_raw_spin_lock+0x177/0x190 [ 5404.199507] _raw_spin_lock+0x21/0x30 [ 5404.201527] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5404.207532] process_one_work+0x2c8/0x700 [ 5404.209192] worker_thread+0x296/0x6e0 [ 5404.210744] ? rescuer_thread+0x570/0x570 [ 5404.212491] kthread+0x1d1/0x200 [ 5404.215316] ? set_kthread_struct+0x70/0x70 [ 5404.217133] ret_from_fork+0x1f/0x30 [ 5408.054711] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5408.064430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5408.091511] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5408.096649] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5408.101411] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5408.103917] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5408.110836] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5408.114494] RAX: 00000000000060e4 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5408.117822] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5408.121219] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5408.123765] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5408.126637] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5408.129379] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5408.132854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5408.135322] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5408.138110] Call Trace: [ 5408.138975] [ 5408.140027] ? show_regs.cold.9+0x22/0x2f [ 5408.141948] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5408.143814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5408.145637] ? watchdog+0x40/0x40 [ 5408.147397] ? hrtimer_interrupt+0x128/0x2c0 [ 5408.148991] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5408.151049] ? apic_timer_interrupt+0xf/0x20 [ 5408.152578] [ 5408.153467] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5408.155429] do_raw_spin_lock+0x177/0x190 [ 5408.158123] _raw_spin_lock+0x21/0x30 [ 5408.160468] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5408.164968] ? do_raw_spin_unlock+0x75/0x190 [ 5408.167465] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5408.170395] ? do_wait_intr+0xf0/0xf0 [ 5408.175402] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5408.178148] mdc_close+0x53b/0xd70 [mdc] [ 5408.183700] lmv_close+0x18d/0x490 [lmv] [ 5408.186022] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5408.189388] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5408.194613] ll_md_close+0x274/0x800 [lustre] [ 5408.200293] ll_file_release+0x128/0xc50 [lustre] [ 5408.203030] __fput+0x10d/0x440 [ 5408.204453] ____fput+0x12/0x20 [ 5408.205895] task_work_run+0xc6/0x110 [ 5408.208196] exit_to_usermode_loop+0x1dd/0x1f0 [ 5408.211914] do_syscall_64+0x3d6/0x440 [ 5408.215101] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5408.217138] RIP: 0033:0x7fdc0afbfcd8 [ 5408.219346] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5408.234910] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5408.241473] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5408.248940] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5408.255636] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5408.259194] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5408.264135] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5412.048023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5412.050698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5412.075696] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5412.083622] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5412.089087] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5412.092974] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5412.105269] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5412.109420] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5412.112965] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5412.116124] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5412.121531] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5412.123951] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5412.129920] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5412.135727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5412.138431] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5412.140924] Call Trace: [ 5412.145162] [ 5412.146326] ? show_regs.cold.9+0x22/0x2f [ 5412.148495] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5412.150756] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5412.153134] ? watchdog+0x40/0x40 [ 5412.154679] ? hrtimer_interrupt+0x128/0x2c0 [ 5412.156581] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5412.158526] ? apic_timer_interrupt+0xf/0x20 [ 5412.160414] [ 5412.161282] ? native_safe_halt+0x2e/0x50 [ 5412.166514] kvm_wait+0xcd/0xe0 [ 5412.167833] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5412.174069] ? dump_stack+0xf6/0x10e [ 5412.175720] do_raw_spin_lock+0x177/0x190 [ 5412.177384] _raw_spin_lock+0x21/0x30 [ 5412.179109] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5412.188071] obd_mod_free+0x46/0x140 [ptlrpc] [ 5412.193348] mdc_commit_open+0x91/0xb0 [mdc] [ 5412.198569] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5412.203467] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5412.209401] after_reply+0xe2c/0x1780 [ptlrpc] [ 5412.211307] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5412.217307] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5412.226507] ? do_wait_intr+0xf0/0xf0 [ 5412.227813] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5412.236344] kthread+0x1d1/0x200 [ 5412.244906] ? set_kthread_struct+0x70/0x70 [ 5412.250730] ret_from_fork+0x1f/0x30 [ 5415.394451] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3390s! [ 5415.398754] Showing busy workqueues and worker pools: [ 5415.411498] workqueue events: flags=0x0 [ 5415.417367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5415.431180] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5415.441492] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5415.460441] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5415.478069] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5415.492374] pending: kfree_rcu_monitor [ 5415.501492] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5415.512831] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5415.544174] workqueue events_long: flags=0x0 [ 5415.557839] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5415.571728] pending: lu_objects_flush [obdclass] [ 5415.581585] workqueue events_unbound: flags=0x2 [ 5415.583958] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5415.594475] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5415.603601] workqueue events_power_efficient: flags=0x80 [ 5415.612090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5415.618025] pending: neigh_periodic_work, check_lifetime [ 5415.627703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5415.632557] pending: do_cache_clean, neigh_periodic_work [ 5415.637151] workqueue mm_percpu_wq: flags=0x8 [ 5415.646498] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5415.658103] pending: lru_add_drain_per_cpu [ 5415.669984] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5415.684912] pending: vmstat_update, lru_add_drain_per_cpu [ 5415.693108] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5415.702823] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5415.713665] workqueue writeback: flags=0x4a [ 5415.725438] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5415.734813] pending: wb_workfn, wb_workfn [ 5415.743093] workqueue xprtiod: flags=0xa [ 5415.748044] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5415.762378] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5415.782239] workqueue ipv6_addrconf: flags=0x40008 [ 5415.797831] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5415.819737] pending: addrconf_verify_work [ 5415.831284] workqueue knbd0-recv: flags=0x1a [ 5415.851837] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5415.865893] in-flight: 655:recv_work [ 5415.868200] workqueue ptlrpc_pinger: flags=0x2 [ 5415.877285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5415.884538] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5415.888577] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3390s workers=4 idle: 36396 38 11188 [ 5415.892517] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5406s workers=2 idle: 658 [ 5428.054798] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 5428.059200] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5428.119738] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5428.125130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5428.132936] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5428.135932] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5428.143840] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5428.146064] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5428.148727] RDX: 00000000000020ed RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5428.152649] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5428.158297] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5428.162563] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5428.166451] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5428.172204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5428.174658] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5428.181460] Call Trace: [ 5428.182494] [ 5428.183828] ? show_regs.cold.9+0x22/0x2f [ 5428.186743] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5428.189200] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5428.191861] ? watchdog+0x40/0x40 [ 5428.199006] ? hrtimer_interrupt+0x128/0x2c0 [ 5428.201735] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5428.204636] ? apic_timer_interrupt+0xf/0x20 [ 5428.206907] [ 5428.208288] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5428.211196] do_raw_spin_lock+0x177/0x190 [ 5428.213204] _raw_spin_lock+0x21/0x30 [ 5428.215279] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5428.217854] ? do_raw_spin_unlock+0x75/0x190 [ 5428.220025] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5428.222773] ? do_wait_intr+0xf0/0xf0 [ 5428.224805] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5428.227453] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5428.230067] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5428.233121] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5428.235806] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5428.238660] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5428.240965] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5428.243182] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5428.247317] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5428.253437] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5428.256626] lmv_intent_open+0x338/0x1150 [lmv] [ 5428.259822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5428.262569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5428.266498] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5428.270022] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5428.272161] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5428.275259] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5428.277863] ? do_raw_spin_unlock+0x75/0x190 [ 5428.279805] ? _raw_spin_unlock+0x12/0x30 [ 5428.281928] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5428.284464] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5428.286698] ll_atomic_open+0x36a/0xd70 [lustre] [ 5428.288461] lookup_open+0x356/0xb40 [ 5428.290320] do_last+0x313/0xfc0 [ 5428.295696] ? nd_jump_root+0xe5/0x160 [ 5428.297704] ? path_init+0x437/0x520 [ 5428.299884] path_openat+0xf7/0x500 [ 5428.301916] ? do_raw_spin_unlock+0x75/0x190 [ 5428.303862] do_filp_open+0x99/0x140 [ 5428.306370] ? getname_flags+0x6e/0x330 [ 5428.308200] ? __check_object_size+0xff/0x256 [ 5428.310588] ? do_raw_spin_unlock+0x75/0x190 [ 5428.315898] ? _raw_spin_unlock+0x12/0x30 [ 5428.318705] do_sys_openat2+0x2b4/0x410 [ 5428.321287] do_sys_open+0x73/0xa0 [ 5428.323197] __x64_sys_openat+0x24/0x30 [ 5428.325240] do_syscall_64+0xc1/0x440 [ 5428.326951] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5428.329554] RIP: 0033:0x7fdc0afbf332 [ 5428.331279] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5428.340862] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5428.343780] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5428.347424] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5428.350670] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5428.354733] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5428.357740] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5432.051021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5432.055397] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5432.101021] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5432.115893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5432.123011] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5432.128561] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5432.138201] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5432.150907] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5432.159986] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5432.168803] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5432.172243] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5432.181268] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5432.191197] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5432.194570] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5432.203455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5432.213158] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5432.220590] Call Trace: [ 5432.221386] [ 5432.224384] ? show_regs.cold.9+0x22/0x2f [ 5432.228970] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5432.235949] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5432.240095] ? watchdog+0x40/0x40 [ 5432.242041] ? hrtimer_interrupt+0x128/0x2c0 [ 5432.253238] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5432.261810] ? apic_timer_interrupt+0xf/0x20 [ 5432.263875] [ 5432.269312] ? native_safe_halt+0x2e/0x50 [ 5432.272900] kvm_wait+0xcd/0xe0 [ 5432.275221] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5432.281192] do_raw_spin_lock+0x177/0x190 [ 5432.285022] _raw_spin_lock+0x21/0x30 [ 5432.287476] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5432.292442] process_one_work+0x2c8/0x700 [ 5432.298467] worker_thread+0x296/0x6e0 [ 5432.303196] ? rescuer_thread+0x570/0x570 [ 5432.310226] kthread+0x1d1/0x200 [ 5432.314272] ? set_kthread_struct+0x70/0x70 [ 5432.317711] ret_from_fork+0x1f/0x30 [ 5436.055392] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5436.060982] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5436.079473] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5436.084927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5436.087846] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5436.090346] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5436.099156] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5436.102360] RAX: 0000000000005813 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5436.106037] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5436.109029] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5436.114374] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5436.118009] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5436.123878] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5436.132220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5436.139369] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5436.147875] Call Trace: [ 5436.149112] [ 5436.149884] ? show_regs.cold.9+0x22/0x2f [ 5436.155782] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5436.157516] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5436.165601] ? watchdog+0x40/0x40 [ 5436.171436] ? hrtimer_interrupt+0x128/0x2c0 [ 5436.173127] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5436.175917] ? apic_timer_interrupt+0xf/0x20 [ 5436.178077] [ 5436.179028] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5436.181344] do_raw_spin_lock+0x177/0x190 [ 5436.183110] _raw_spin_lock+0x21/0x30 [ 5436.184500] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5436.187036] ? do_raw_spin_unlock+0x75/0x190 [ 5436.188498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5436.190725] ? do_wait_intr+0xf0/0xf0 [ 5436.192129] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5436.194097] mdc_close+0x53b/0xd70 [mdc] [ 5436.195521] lmv_close+0x18d/0x490 [lmv] [ 5436.197194] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5436.199546] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5436.201617] ll_md_close+0x274/0x800 [lustre] [ 5436.203652] ll_file_release+0x128/0xc50 [lustre] [ 5436.205755] __fput+0x10d/0x440 [ 5436.207049] ____fput+0x12/0x20 [ 5436.208102] task_work_run+0xc6/0x110 [ 5436.209623] exit_to_usermode_loop+0x1dd/0x1f0 [ 5436.211478] do_syscall_64+0x3d6/0x440 [ 5436.218873] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5436.221455] RIP: 0033:0x7fdc0afbfcd8 [ 5436.229548] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5436.246664] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5436.252402] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5436.258075] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5436.267870] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5436.274360] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5436.277126] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5440.053483] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5440.065177] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5440.130454] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5440.136706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5440.153247] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5440.164367] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5440.176069] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5440.179939] RAX: 0000000000006719 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5440.184736] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5440.189275] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5440.201791] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5440.206494] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5440.219078] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5440.232493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5440.238702] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5440.247444] Call Trace: [ 5440.248714] [ 5440.250115] ? show_regs.cold.9+0x22/0x2f [ 5440.252689] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5440.258109] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5440.260996] ? watchdog+0x40/0x40 [ 5440.263209] ? hrtimer_interrupt+0x128/0x2c0 [ 5440.265775] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5440.274494] ? apic_timer_interrupt+0xf/0x20 [ 5440.281524] [ 5440.283358] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5440.287150] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5440.292585] ? dump_stack+0xf6/0x10e [ 5440.294365] do_raw_spin_lock+0x177/0x190 [ 5440.296736] _raw_spin_lock+0x21/0x30 [ 5440.297995] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5440.302003] obd_mod_free+0x46/0x140 [ptlrpc] [ 5440.304780] mdc_commit_open+0x91/0xb0 [mdc] [ 5440.306968] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5440.309350] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5440.312689] after_reply+0xe2c/0x1780 [ptlrpc] [ 5440.318507] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5440.322000] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5440.326761] ? do_wait_intr+0xf0/0xf0 [ 5440.327727] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5440.333725] kthread+0x1d1/0x200 [ 5440.334852] ? set_kthread_struct+0x70/0x70 [ 5440.338716] ret_from_fork+0x1f/0x30 [ 5446.113545] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3421s! [ 5446.123694] Showing busy workqueues and worker pools: [ 5446.125977] workqueue events: flags=0x0 [ 5446.129637] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5446.137726] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5446.148649] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5446.155384] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5446.176214] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5446.181707] pending: kfree_rcu_monitor [ 5446.183825] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5446.193088] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5446.204283] workqueue events_long: flags=0x0 [ 5446.205878] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5446.219520] pending: lu_objects_flush [obdclass] [ 5446.225718] workqueue events_unbound: flags=0x2 [ 5446.228436] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5446.235128] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5446.243870] workqueue events_power_efficient: flags=0x80 [ 5446.247538] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5446.252022] pending: neigh_periodic_work, check_lifetime [ 5446.257642] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5446.260754] pending: do_cache_clean, neigh_periodic_work [ 5446.272356] workqueue mm_percpu_wq: flags=0x8 [ 5446.277831] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5446.284760] pending: lru_add_drain_per_cpu [ 5446.289678] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5446.298392] pending: vmstat_update, lru_add_drain_per_cpu [ 5446.302530] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5446.307387] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5446.313701] workqueue writeback: flags=0x4a [ 5446.315726] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5446.323516] pending: wb_workfn, wb_workfn [ 5446.329958] workqueue xprtiod: flags=0xa [ 5446.332344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5446.341041] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5446.345899] workqueue ipv6_addrconf: flags=0x40008 [ 5446.348761] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5446.355359] pending: addrconf_verify_work [ 5446.357796] workqueue knbd0-recv: flags=0x1a [ 5446.363561] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5446.373634] in-flight: 655:recv_work [ 5446.380150] workqueue ptlrpc_pinger: flags=0x2 [ 5446.382563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5446.390756] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5446.395448] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3421s workers=4 idle: 36396 38 11188 [ 5446.401180] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5436s workers=2 idle: 658 [ 5456.052058] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 5456.055147] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5456.072887] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5456.086561] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5456.089277] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5456.101499] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5456.118071] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5456.124122] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5456.129802] RDX: 0000000000001f1c RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5456.132989] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5456.142810] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5456.148955] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5456.157115] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5456.166078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5456.170195] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5456.178207] Call Trace: [ 5456.180567] [ 5456.182331] ? show_regs.cold.9+0x22/0x2f [ 5456.184548] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5456.188399] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5456.190199] ? watchdog+0x40/0x40 [ 5456.192697] ? hrtimer_interrupt+0x128/0x2c0 [ 5456.196501] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5456.200563] ? apic_timer_interrupt+0xf/0x20 [ 5456.203564] [ 5456.205820] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5456.209836] do_raw_spin_lock+0x177/0x190 [ 5456.211958] _raw_spin_lock+0x21/0x30 [ 5456.213001] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5456.217787] ? do_raw_spin_unlock+0x75/0x190 [ 5456.221917] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5456.228275] ? do_wait_intr+0xf0/0xf0 [ 5456.231611] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5456.233537] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5456.238185] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5456.244873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5456.250534] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5456.254554] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5456.255707] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5456.257168] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5456.260090] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5456.263686] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5456.265982] lmv_intent_open+0x338/0x1150 [lmv] [ 5456.269534] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5456.271495] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5456.273451] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5456.275479] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5456.277549] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5456.281502] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5456.286807] ? do_raw_spin_unlock+0x75/0x190 [ 5456.289863] ? _raw_spin_unlock+0x12/0x30 [ 5456.293064] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5456.295723] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5456.299856] ll_atomic_open+0x36a/0xd70 [lustre] [ 5456.302647] lookup_open+0x356/0xb40 [ 5456.303871] do_last+0x313/0xfc0 [ 5456.305347] ? nd_jump_root+0xe5/0x160 [ 5456.308297] ? path_init+0x437/0x520 [ 5456.309227] path_openat+0xf7/0x500 [ 5456.311424] ? do_raw_spin_unlock+0x75/0x190 [ 5456.313700] do_filp_open+0x99/0x140 [ 5456.315246] ? getname_flags+0x6e/0x330 [ 5456.316994] ? __check_object_size+0xff/0x256 [ 5456.319021] ? do_raw_spin_unlock+0x75/0x190 [ 5456.321024] ? _raw_spin_unlock+0x12/0x30 [ 5456.322817] do_sys_openat2+0x2b4/0x410 [ 5456.324558] do_sys_open+0x73/0xa0 [ 5456.326451] __x64_sys_openat+0x24/0x30 [ 5456.330463] do_syscall_64+0xc1/0x440 [ 5456.333166] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5456.335214] RIP: 0033:0x7fdc0afbf332 [ 5456.337067] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5456.345716] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5456.348731] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5456.352573] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5456.355391] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5456.358342] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5456.361543] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5460.051056] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 5460.054183] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5460.079488] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5460.084751] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5460.089527] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5460.093007] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5460.095316] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5460.104412] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5460.108268] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5460.111372] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5460.114878] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5460.119244] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5460.122560] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5460.125890] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5460.130655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5460.134618] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5460.138041] Call Trace: [ 5460.138976] [ 5460.139735] ? show_regs.cold.9+0x22/0x2f [ 5460.142800] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5460.145601] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5460.148283] ? watchdog+0x40/0x40 [ 5460.149578] ? hrtimer_interrupt+0x128/0x2c0 [ 5460.151529] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5460.153763] ? apic_timer_interrupt+0xf/0x20 [ 5460.155837] [ 5460.156914] ? native_safe_halt+0x2e/0x50 [ 5460.158778] kvm_wait+0xcd/0xe0 [ 5460.159925] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5460.162373] do_raw_spin_lock+0x177/0x190 [ 5460.163855] _raw_spin_lock+0x21/0x30 [ 5460.165660] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5460.168518] process_one_work+0x2c8/0x700 [ 5460.170384] worker_thread+0x296/0x6e0 [ 5460.172568] ? rescuer_thread+0x570/0x570 [ 5460.174422] kthread+0x1d1/0x200 [ 5460.176052] ? set_kthread_struct+0x70/0x70 [ 5460.177750] ret_from_fork+0x1f/0x30 [ 5464.056069] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5464.061489] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5464.101910] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5464.112421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5464.126797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5464.134151] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5464.150151] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [ 5464.154495] RAX: 0000000000006f00 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5464.159498] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5464.162695] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5464.166925] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5464.170748] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5464.174943] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5464.179478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5464.182504] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5464.186123] Call Trace: [ 5464.187203] [ 5464.188427] ? show_regs.cold.9+0x22/0x2f [ 5464.191216] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5464.194246] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5464.197527] ? watchdog+0x40/0x40 [ 5464.200541] ? hrtimer_interrupt+0x128/0x2c0 [ 5464.206467] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5464.211260] ? apic_timer_interrupt+0xf/0x20 [ 5464.213996] [ 5464.215898] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5464.223295] do_raw_spin_lock+0x177/0x190 [ 5464.227263] _raw_spin_lock+0x21/0x30 [ 5464.229771] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5464.233537] ? do_raw_spin_unlock+0x75/0x190 [ 5464.236275] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5464.239044] ? do_wait_intr+0xf0/0xf0 [ 5464.241436] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5464.246173] mdc_close+0x53b/0xd70 [mdc] [ 5464.248471] lmv_close+0x18d/0x490 [lmv] [ 5464.250888] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5464.254131] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5464.256901] ll_md_close+0x274/0x800 [lustre] [ 5464.260036] ll_file_release+0x128/0xc50 [lustre] [ 5464.263513] __fput+0x10d/0x440 [ 5464.265277] ____fput+0x12/0x20 [ 5464.267143] task_work_run+0xc6/0x110 [ 5464.269019] exit_to_usermode_loop+0x1dd/0x1f0 [ 5464.271330] do_syscall_64+0x3d6/0x440 [ 5464.273461] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5464.276005] RIP: 0033:0x7fdc0afbfcd8 [ 5464.280227] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5464.292664] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5464.296540] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5464.300194] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5464.303903] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5464.307390] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5464.311418] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5472.049627] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 5472.061755] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5472.097967] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5472.102979] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5472.110588] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5472.118327] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5472.134399] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5472.139066] RAX: 0000000000007c86 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5472.146346] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5472.153154] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5472.160414] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5472.165171] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5472.173333] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5472.185223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5472.190698] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5472.197525] Call Trace: [ 5472.199022] [ 5472.200273] ? show_regs.cold.9+0x22/0x2f [ 5472.202771] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5472.210037] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5472.211655] ? watchdog+0x40/0x40 [ 5472.213303] ? hrtimer_interrupt+0x128/0x2c0 [ 5472.215140] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5472.223096] ? apic_timer_interrupt+0xf/0x20 [ 5472.226709] [ 5472.230535] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5472.236544] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5472.242898] ? dump_stack+0xf6/0x10e [ 5472.244959] do_raw_spin_lock+0x177/0x190 [ 5472.250835] _raw_spin_lock+0x21/0x30 [ 5472.256065] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5472.267723] obd_mod_free+0x46/0x140 [ptlrpc] [ 5472.273012] mdc_commit_open+0x91/0xb0 [mdc] [ 5472.275690] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5472.282209] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5472.286046] after_reply+0xe2c/0x1780 [ptlrpc] [ 5472.288340] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5472.295459] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5472.298706] ? do_wait_intr+0xf0/0xf0 [ 5472.304141] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5472.305922] kthread+0x1d1/0x200 [ 5472.308528] ? set_kthread_struct+0x70/0x70 [ 5472.310346] ret_from_fork+0x1f/0x30 [ 5476.836229] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3451s! [ 5476.838953] Showing busy workqueues and worker pools: [ 5476.841135] workqueue events: flags=0x0 [ 5476.842642] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5476.845007] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5476.847842] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5476.850492] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5476.854383] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5476.857028] pending: kfree_rcu_monitor [ 5476.858785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5476.861362] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5476.866307] workqueue events_long: flags=0x0 [ 5476.867829] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5476.870358] pending: lu_objects_flush [obdclass] [ 5476.872968] workqueue events_unbound: flags=0x2 [ 5476.874869] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5476.877382] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5476.881907] workqueue events_power_efficient: flags=0x80 [ 5476.883863] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5476.886484] pending: neigh_periodic_work, check_lifetime [ 5476.888759] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5476.891365] pending: do_cache_clean, neigh_periodic_work [ 5476.893642] workqueue mm_percpu_wq: flags=0x8 [ 5476.895443] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5476.897982] pending: lru_add_drain_per_cpu [ 5476.899779] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5476.902484] pending: vmstat_update, lru_add_drain_per_cpu [ 5476.904738] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5476.907351] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5476.909941] workqueue writeback: flags=0x4a [ 5476.911384] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5476.914003] pending: wb_workfn, wb_workfn [ 5476.915811] workqueue xprtiod: flags=0xa [ 5476.917279] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5476.919830] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5476.922797] workqueue ipv6_addrconf: flags=0x40008 [ 5476.924688] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5476.927254] pending: addrconf_verify_work [ 5476.928648] workqueue knbd0-recv: flags=0x1a [ 5476.930071] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5476.932376] in-flight: 655:recv_work [ 5476.934026] workqueue ptlrpc_pinger: flags=0x2 [ 5476.935954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5476.938206] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5476.940871] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3452s workers=4 idle: 36396 38 11188 [ 5476.944177] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5467s workers=2 idle: 658 [ 5484.053124] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5484.064929] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5484.145275] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5484.162242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5484.176340] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5484.183902] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5484.223251] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5484.237235] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5484.247843] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5484.260952] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5484.267241] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5484.280775] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5484.293785] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5484.310757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5484.315805] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5484.326225] Call Trace: [ 5484.332319] [ 5484.335838] ? show_regs.cold.9+0x22/0x2f [ 5484.346473] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5484.348190] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5484.359369] ? watchdog+0x40/0x40 [ 5484.366089] ? hrtimer_interrupt+0x128/0x2c0 [ 5484.371167] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5484.372726] ? apic_timer_interrupt+0xf/0x20 [ 5484.378385] [ 5484.384163] ? native_safe_halt+0x2e/0x50 [ 5484.390390] kvm_wait+0xcd/0xe0 [ 5484.401221] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5484.410852] do_raw_spin_lock+0x177/0x190 [ 5484.412212] _raw_spin_lock+0x21/0x30 [ 5484.423572] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5484.435220] ? do_raw_spin_unlock+0x75/0x190 [ 5484.436745] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5484.438588] ? do_wait_intr+0xf0/0xf0 [ 5484.458259] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5484.473344] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5484.475275] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5484.491684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5484.497283] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5484.503847] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5484.510265] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5484.517715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5484.535936] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5484.538038] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5484.563704] lmv_intent_open+0x338/0x1150 [lmv] [ 5484.571661] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5484.574162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5484.584787] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5484.595474] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5484.597057] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5484.609833] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5484.622735] ? do_raw_spin_unlock+0x75/0x190 [ 5484.629583] ? _raw_spin_unlock+0x12/0x30 [ 5484.632897] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5484.638155] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5484.641349] ll_atomic_open+0x36a/0xd70 [lustre] [ 5484.645799] lookup_open+0x356/0xb40 [ 5484.647875] do_last+0x313/0xfc0 [ 5484.649384] ? nd_jump_root+0xe5/0x160 [ 5484.650991] ? path_init+0x437/0x520 [ 5484.655762] path_openat+0xf7/0x500 [ 5484.666200] ? do_raw_spin_unlock+0x75/0x190 [ 5484.670383] do_filp_open+0x99/0x140 [ 5484.676677] ? getname_flags+0x6e/0x330 [ 5484.679279] ? __check_object_size+0xff/0x256 [ 5484.685967] ? do_raw_spin_unlock+0x75/0x190 [ 5484.689990] ? _raw_spin_unlock+0x12/0x30 [ 5484.691975] do_sys_openat2+0x2b4/0x410 [ 5484.694211] do_sys_open+0x73/0xa0 [ 5484.695685] __x64_sys_openat+0x24/0x30 [ 5484.697667] do_syscall_64+0xc1/0x440 [ 5484.699152] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5484.702789] RIP: 0033:0x7fdc0afbf332 [ 5484.704214] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5484.722174] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5484.730800] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5484.734459] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5484.744957] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5484.751405] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5484.755619] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5488.051099] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 5488.057338] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5488.105033] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5488.112080] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5488.118625] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5488.121603] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5488.123519] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5488.130883] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5488.134055] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5488.136834] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5488.139691] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5488.142988] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5488.146164] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5488.149400] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5488.152656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5488.155481] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5488.158598] Call Trace: [ 5488.159623] [ 5488.160322] ? show_regs.cold.9+0x22/0x2f [ 5488.161614] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5488.163519] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5488.165497] ? watchdog+0x40/0x40 [ 5488.166851] ? hrtimer_interrupt+0x128/0x2c0 [ 5488.168599] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5488.170628] ? apic_timer_interrupt+0xf/0x20 [ 5488.172541] [ 5488.173512] ? native_safe_halt+0x2e/0x50 [ 5488.175200] kvm_wait+0xcd/0xe0 [ 5488.176688] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5488.178968] do_raw_spin_lock+0x177/0x190 [ 5488.180621] _raw_spin_lock+0x21/0x30 [ 5488.182155] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5488.184803] process_one_work+0x2c8/0x700 [ 5488.186489] worker_thread+0x296/0x6e0 [ 5488.188047] ? rescuer_thread+0x570/0x570 [ 5488.189748] kthread+0x1d1/0x200 [ 5488.191104] ? set_kthread_struct+0x70/0x70 [ 5488.193139] ret_from_fork+0x1f/0x30 [ 5492.054034] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5492.056757] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5492.094086] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5492.101374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5492.109152] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5492.111556] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5492.127329] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5492.133790] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5492.137645] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5492.141543] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5492.145247] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5492.148596] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5492.156708] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5492.164642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5492.168444] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5492.174196] Call Trace: [ 5492.177708] [ 5492.179195] ? show_regs.cold.9+0x22/0x2f [ 5492.181173] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5492.186144] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5492.189008] ? watchdog+0x40/0x40 [ 5492.190688] ? hrtimer_interrupt+0x128/0x2c0 [ 5492.193473] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5492.195655] ? apic_timer_interrupt+0xf/0x20 [ 5492.197830] [ 5492.200618] ? native_safe_halt+0x2e/0x50 [ 5492.202559] kvm_wait+0xcd/0xe0 [ 5492.204648] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5492.206550] do_raw_spin_lock+0x177/0x190 [ 5492.208679] _raw_spin_lock+0x21/0x30 [ 5492.210465] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5492.212803] ? do_raw_spin_unlock+0x75/0x190 [ 5492.217722] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5492.220809] ? do_wait_intr+0xf0/0xf0 [ 5492.223141] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5492.225469] mdc_close+0x53b/0xd70 [mdc] [ 5492.227374] lmv_close+0x18d/0x490 [lmv] [ 5492.229278] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5492.233855] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5492.235878] ll_md_close+0x274/0x800 [lustre] [ 5492.238298] ll_file_release+0x128/0xc50 [lustre] [ 5492.240249] __fput+0x10d/0x440 [ 5492.241160] ____fput+0x12/0x20 [ 5492.242902] task_work_run+0xc6/0x110 [ 5492.244858] exit_to_usermode_loop+0x1dd/0x1f0 [ 5492.247209] do_syscall_64+0x3d6/0x440 [ 5492.248678] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5492.250677] RIP: 0033:0x7fdc0afbfcd8 [ 5492.254135] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5492.263209] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5492.272783] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5492.276704] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5492.279950] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5492.285789] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5492.290356] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5499.424320] rcu: INFO: rcu_sched self-detected stall on CPU [ 5499.433200] rcu: 0-...!: (2063776 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 5499.441534] (t=3480081 jiffies g=270761 q=3511) [ 5499.453661] rcu: rcu_sched kthread starved for 3475013 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 5499.460492] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5499.470850] rcu: RCU grace-period kthread stack dump: [ 5499.476034] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5499.486072] Call Trace: [ 5499.487978] __schedule+0x351/0xcb0 [ 5499.491428] schedule+0xc0/0x180 [ 5499.494679] schedule_timeout+0xb4/0x190 [ 5499.496027] ? do_raw_spin_unlock+0x75/0x190 [ 5499.502364] ? __next_timer_interrupt+0x160/0x160 [ 5499.505645] ? prepare_to_swait_event+0xf3/0x140 [ 5499.508093] rcu_gp_kthread+0x8dd/0x15b0 [ 5499.510177] ? force_qs_rnp+0x2c0/0x2c0 [ 5499.511452] kthread+0x1d1/0x200 [ 5499.513283] ? set_kthread_struct+0x70/0x70 [ 5499.515397] ret_from_fork+0x1f/0x30 [ 5499.521392] rcu: Stack dump where RCU GP kthread last ran: [ 5499.524044] Sending NMI from CPU 0 to CPUs 1: [ 5499.533768] NMI backtrace for cpu 1 [ 5499.533777] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5499.533780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5499.533782] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5499.533784] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5499.533786] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5499.533788] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 [ 5499.533790] RAX: 0000000000007b0f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5499.533791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5499.533793] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5499.533794] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5499.533795] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5499.533796] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5499.533798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5499.533799] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5499.533800] Call Trace: [ 5499.533801] [ 5499.533802] ? show_regs.cold.9+0x22/0x2f [ 5499.533803] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5499.533804] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5499.533806] ? nmi_handle+0xbd/0x210 [ 5499.533807] ? default_do_nmi+0x4b/0x1b0 [ 5499.533808] ? do_nmi+0x280/0x380 [ 5499.533808] ? end_repeat_nmi+0x16/0x1f [ 5499.533810] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5499.533811] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5499.533812] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5499.533814] [ 5499.533814] do_raw_spin_lock+0x177/0x190 [ 5499.533815] _raw_spin_lock+0x21/0x30 [ 5499.533816] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5499.533817] process_one_work+0x2c8/0x700 [ 5499.533819] worker_thread+0x296/0x6e0 [ 5499.533820] ? rescuer_thread+0x570/0x570 [ 5499.533821] kthread+0x1d1/0x200 [ 5499.533822] ? set_kthread_struct+0x70/0x70 [ 5499.533823] ret_from_fork+0x1f/0x30 [ 5499.536670] NMI backtrace for cpu 0 [ 5499.678736] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5499.691309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5499.700730] Call Trace: [ 5499.702375] [ 5499.708069] ? dump_stack+0xbb/0x10e [ 5499.713525] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5499.717110] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5499.721103] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5499.725219] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5499.730768] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5499.735407] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5499.738310] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5499.741481] ? update_process_times+0x6d/0xc0 [ 5499.742760] ? tick_sched_handle.isra.18+0x27/0x90 [ 5499.747424] ? tick_sched_timer+0x53/0xb0 [ 5499.749537] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5499.752258] ? tick_sched_handle.isra.18+0x90/0x90 [ 5499.756320] ? hrtimer_interrupt+0x128/0x2c0 [ 5499.758393] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5499.761997] ? apic_timer_interrupt+0xf/0x20 [ 5499.763195] [ 5499.766725] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5499.769075] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5499.771409] ? dump_stack+0xf6/0x10e [ 5499.773944] ? do_raw_spin_lock+0x177/0x190 [ 5499.776408] ? _raw_spin_lock+0x21/0x30 [ 5499.780317] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5499.786510] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5499.789020] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5499.792047] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5499.801212] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5499.812227] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5499.818310] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5499.823696] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5499.825266] ? do_wait_intr+0xf0/0xf0 [ 5499.830553] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5499.832169] ? kthread+0x1d1/0x200 [ 5499.833081] ? set_kthread_struct+0x70/0x70 [ 5499.837580] ? ret_from_fork+0x1f/0x30 [ 5507.040111] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3482s! [ 5507.050841] Showing busy workqueues and worker pools: [ 5507.054709] workqueue events: flags=0x0 [ 5507.056397] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5507.059163] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5507.062661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5507.076368] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5507.089428] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5507.096156] pending: kfree_rcu_monitor [ 5507.100831] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5507.107376] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5507.130701] workqueue events_long: flags=0x0 [ 5507.136364] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5507.141280] pending: lu_objects_flush [obdclass] [ 5507.144157] workqueue events_unbound: flags=0x2 [ 5507.146266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5507.154725] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5507.160831] workqueue events_power_efficient: flags=0x80 [ 5507.175648] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5507.182921] pending: neigh_periodic_work, check_lifetime [ 5507.194687] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5507.204692] pending: do_cache_clean, neigh_periodic_work [ 5507.207280] workqueue mm_percpu_wq: flags=0x8 [ 5507.209377] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5507.213858] pending: lru_add_drain_per_cpu [ 5507.227166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5507.230118] pending: vmstat_update, lru_add_drain_per_cpu [ 5507.240768] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5507.243248] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5507.246098] workqueue writeback: flags=0x4a [ 5507.247885] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5507.250635] pending: wb_workfn, wb_workfn [ 5507.252648] workqueue xprtiod: flags=0xa [ 5507.254871] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5507.257452] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5507.260484] workqueue ipv6_addrconf: flags=0x40008 [ 5507.261738] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5507.264506] pending: addrconf_verify_work [ 5507.265971] workqueue knbd0-recv: flags=0x1a [ 5507.269597] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5507.272261] in-flight: 655:recv_work [ 5507.276643] workqueue ptlrpc_pinger: flags=0x2 [ 5507.279953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5507.282049] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5507.289383] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3482s workers=4 idle: 36396 38 11188 [ 5507.296848] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5497s workers=2 idle: 658 [ 5512.052050] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5512.055460] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5512.074203] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5512.078351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5512.081736] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5512.084246] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5512.092235] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5512.095565] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5512.099922] RDX: 00000000000063ab RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5512.102733] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5512.105701] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5512.108655] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5512.111467] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5512.114612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5512.116943] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5512.119875] Call Trace: [ 5512.120850] [ 5512.121653] ? show_regs.cold.9+0x22/0x2f [ 5512.123466] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5512.125552] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5512.127503] ? watchdog+0x40/0x40 [ 5512.128697] ? hrtimer_interrupt+0x128/0x2c0 [ 5512.130552] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5512.132505] ? apic_timer_interrupt+0xf/0x20 [ 5512.134285] [ 5512.135377] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5512.137567] do_raw_spin_lock+0x177/0x190 [ 5512.139099] _raw_spin_lock+0x21/0x30 [ 5512.140631] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5512.142755] ? do_raw_spin_unlock+0x75/0x190 [ 5512.144677] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5512.146799] ? do_wait_intr+0xf0/0xf0 [ 5512.148292] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5512.150762] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5512.153370] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5512.156276] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5512.158823] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5512.161031] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5512.162490] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5512.164688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5512.167173] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5512.169844] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5512.172422] lmv_intent_open+0x338/0x1150 [lmv] [ 5512.174517] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5512.177315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5512.179702] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5512.181739] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5512.183998] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5512.186417] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5512.188415] ? do_raw_spin_unlock+0x75/0x190 [ 5512.190574] ? _raw_spin_unlock+0x12/0x30 [ 5512.193226] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5512.196392] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5512.199288] ll_atomic_open+0x36a/0xd70 [lustre] [ 5512.201998] lookup_open+0x356/0xb40 [ 5512.204015] do_last+0x313/0xfc0 [ 5512.206440] ? nd_jump_root+0xe5/0x160 [ 5512.208128] ? path_init+0x437/0x520 [ 5512.209752] path_openat+0xf7/0x500 [ 5512.211906] ? do_raw_spin_unlock+0x75/0x190 [ 5512.214046] do_filp_open+0x99/0x140 [ 5512.215788] ? getname_flags+0x6e/0x330 [ 5512.218072] ? __check_object_size+0xff/0x256 [ 5512.220325] ? do_raw_spin_unlock+0x75/0x190 [ 5512.222225] ? _raw_spin_unlock+0x12/0x30 [ 5512.224170] do_sys_openat2+0x2b4/0x410 [ 5512.226447] do_sys_open+0x73/0xa0 [ 5512.228228] __x64_sys_openat+0x24/0x30 [ 5512.229422] do_syscall_64+0xc1/0x440 [ 5512.231114] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5512.233431] RIP: 0033:0x7fdc0afbf332 [ 5512.234826] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5512.242144] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5512.245516] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5512.249283] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5512.252742] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5512.255788] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5512.259314] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5520.054023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5520.057418] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5520.077268] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5520.082004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5520.085605] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5520.088356] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5520.095795] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5520.098434] RAX: 0000000000006789 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5520.101586] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5520.104644] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5520.107306] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5520.110277] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5520.113113] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5520.116363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5520.119052] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5520.121932] Call Trace: [ 5520.123009] [ 5520.124184] ? show_regs.cold.9+0x22/0x2f [ 5520.126002] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5520.128118] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5520.130487] ? watchdog+0x40/0x40 [ 5520.132418] ? hrtimer_interrupt+0x128/0x2c0 [ 5520.134361] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5520.136493] ? apic_timer_interrupt+0xf/0x20 [ 5520.138412] [ 5520.139394] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5520.142415] do_raw_spin_lock+0x177/0x190 [ 5520.144335] _raw_spin_lock+0x21/0x30 [ 5520.145717] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5520.148298] ? do_raw_spin_unlock+0x75/0x190 [ 5520.150176] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5520.152581] ? do_wait_intr+0xf0/0xf0 [ 5520.154321] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5520.156441] mdc_close+0x53b/0xd70 [mdc] [ 5520.158054] lmv_close+0x18d/0x490 [lmv] [ 5520.159971] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5520.162280] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5520.165166] ll_md_close+0x274/0x800 [lustre] [ 5520.167264] ll_file_release+0x128/0xc50 [lustre] [ 5520.169351] __fput+0x10d/0x440 [ 5520.171004] ____fput+0x12/0x20 [ 5520.172325] task_work_run+0xc6/0x110 [ 5520.174183] exit_to_usermode_loop+0x1dd/0x1f0 [ 5520.176973] do_syscall_64+0x3d6/0x440 [ 5520.178712] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5520.181221] RIP: 0033:0x7fdc0afbfcd8 [ 5520.183430] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5520.192677] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5520.195967] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5520.198917] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5520.201815] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5520.204662] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5520.207538] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5524.055825] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5524.066498] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5524.108214] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5524.117562] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5524.122267] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5524.125061] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5524.127614] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5524.136372] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5524.139477] RAX: 0000000000007c4b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5524.144701] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5524.147554] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5524.151264] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5524.153735] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5524.158832] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5524.163170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5524.165649] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5524.168912] Call Trace: [ 5524.169962] [ 5524.170919] ? show_regs.cold.9+0x22/0x2f [ 5524.172919] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5524.175440] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5524.177373] ? watchdog+0x40/0x40 [ 5524.178768] ? hrtimer_interrupt+0x128/0x2c0 [ 5524.180657] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5524.182974] ? apic_timer_interrupt+0xf/0x20 [ 5524.184645] [ 5524.185669] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5524.187990] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5524.190497] do_raw_spin_lock+0x177/0x190 [ 5524.192806] _raw_spin_lock+0x21/0x30 [ 5524.194633] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5524.197570] process_one_work+0x2c8/0x700 [ 5524.199360] worker_thread+0x296/0x6e0 [ 5524.200985] ? rescuer_thread+0x570/0x570 [ 5524.203621] kthread+0x1d1/0x200 [ 5524.205284] ? set_kthread_struct+0x70/0x70 [ 5524.207323] ret_from_fork+0x1f/0x30 [ 5532.048034] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 5532.054781] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5532.090943] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5532.095319] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5532.107832] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5532.114398] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5532.139084] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5532.146744] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5532.150685] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5532.154297] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5532.157894] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5532.160826] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5532.163967] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5532.167973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5532.171805] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5532.175679] Call Trace: [ 5532.177267] [ 5532.178319] ? show_regs.cold.9+0x22/0x2f [ 5532.180689] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5532.183232] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5532.185750] ? watchdog+0x40/0x40 [ 5532.187458] ? hrtimer_interrupt+0x128/0x2c0 [ 5532.189465] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5532.191657] ? apic_timer_interrupt+0xf/0x20 [ 5532.193551] [ 5532.194473] ? native_safe_halt+0x2e/0x50 [ 5532.196421] kvm_wait+0xcd/0xe0 [ 5532.197908] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5532.200416] ? dump_stack+0xf6/0x10e [ 5532.202256] do_raw_spin_lock+0x177/0x190 [ 5532.204455] _raw_spin_lock+0x21/0x30 [ 5532.206103] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5532.216015] obd_mod_free+0x46/0x140 [ptlrpc] [ 5532.221869] mdc_commit_open+0x91/0xb0 [mdc] [ 5532.225249] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5532.229779] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5532.233326] after_reply+0xe2c/0x1780 [ptlrpc] [ 5532.236448] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5532.239468] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5532.241705] ? do_wait_intr+0xf0/0xf0 [ 5532.243002] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5532.245543] kthread+0x1d1/0x200 [ 5532.247436] ? set_kthread_struct+0x70/0x70 [ 5532.253480] ret_from_fork+0x1f/0x30 [ 5537.759935] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3512s! [ 5537.771160] Showing busy workqueues and worker pools: [ 5537.775135] workqueue events: flags=0x0 [ 5537.781609] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5537.787160] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5537.791671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5537.800431] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5537.809357] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5537.815584] pending: kfree_rcu_monitor [ 5537.819667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5537.824149] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5537.837371] workqueue events_long: flags=0x0 [ 5537.841458] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5537.848701] pending: lu_objects_flush [obdclass] [ 5537.852955] workqueue events_unbound: flags=0x2 [ 5537.855802] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5537.866726] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5537.874210] workqueue events_power_efficient: flags=0x80 [ 5537.878505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5537.884753] pending: neigh_periodic_work, check_lifetime [ 5537.887555] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5537.889827] pending: do_cache_clean, neigh_periodic_work [ 5537.902422] workqueue mm_percpu_wq: flags=0x8 [ 5537.906111] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5537.916193] pending: lru_add_drain_per_cpu [ 5537.917699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5537.936841] pending: vmstat_update, lru_add_drain_per_cpu [ 5537.946312] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5537.951062] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5537.954141] workqueue writeback: flags=0x4a [ 5537.956357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5537.964873] pending: wb_workfn, wb_workfn [ 5537.966465] workqueue xprtiod: flags=0xa [ 5537.969707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5537.974660] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5537.986831] workqueue ipv6_addrconf: flags=0x40008 [ 5537.991711] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5538.001930] pending: addrconf_verify_work [ 5538.003611] workqueue knbd0-recv: flags=0x1a [ 5538.010682] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5538.022226] in-flight: 655:recv_work [ 5538.024752] workqueue ptlrpc_pinger: flags=0x2 [ 5538.032461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5538.043403] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5538.053731] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3513s workers=4 idle: 36396 38 11188 [ 5538.065868] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5528s workers=2 idle: 658 [ 5540.058362] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5540.065640] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5540.124098] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5540.130116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5540.143612] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5540.150535] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5540.170240] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5540.176499] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5540.189572] RDX: 00000000000013c0 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5540.199488] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5540.207115] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5540.214095] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5540.221356] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5540.231692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5540.237274] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5540.247905] Call Trace: [ 5540.251264] [ 5540.257926] ? show_regs.cold.9+0x22/0x2f [ 5540.259693] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5540.267448] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5540.269719] ? watchdog+0x40/0x40 [ 5540.274770] ? hrtimer_interrupt+0x128/0x2c0 [ 5540.284741] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5540.286747] ? apic_timer_interrupt+0xf/0x20 [ 5540.292431] [ 5540.293499] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5540.300535] do_raw_spin_lock+0x177/0x190 [ 5540.302620] _raw_spin_lock+0x21/0x30 [ 5540.304351] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5540.309383] ? do_raw_spin_unlock+0x75/0x190 [ 5540.311380] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5540.321420] ? do_wait_intr+0xf0/0xf0 [ 5540.327856] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5540.330430] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5540.335904] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5540.342472] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5540.356576] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5540.365060] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5540.367135] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5540.372705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5540.388835] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5540.398333] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5540.403436] lmv_intent_open+0x338/0x1150 [lmv] [ 5540.405863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5540.417138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5540.422840] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5540.425940] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5540.430678] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5540.438795] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5540.447763] ? do_raw_spin_unlock+0x75/0x190 [ 5540.450799] ? _raw_spin_unlock+0x12/0x30 [ 5540.456096] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5540.460388] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5540.468195] ll_atomic_open+0x36a/0xd70 [lustre] [ 5540.474266] lookup_open+0x356/0xb40 [ 5540.476237] do_last+0x313/0xfc0 [ 5540.477266] ? nd_jump_root+0xe5/0x160 [ 5540.481074] ? path_init+0x437/0x520 [ 5540.482790] path_openat+0xf7/0x500 [ 5540.495580] ? do_raw_spin_unlock+0x75/0x190 [ 5540.497091] do_filp_open+0x99/0x140 [ 5540.501520] ? getname_flags+0x6e/0x330 [ 5540.504783] ? __check_object_size+0xff/0x256 [ 5540.514377] ? do_raw_spin_unlock+0x75/0x190 [ 5540.521740] ? _raw_spin_unlock+0x12/0x30 [ 5540.529751] do_sys_openat2+0x2b4/0x410 [ 5540.542039] do_sys_open+0x73/0xa0 [ 5540.543111] __x64_sys_openat+0x24/0x30 [ 5540.553133] do_syscall_64+0xc1/0x440 [ 5540.554556] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5540.563874] RIP: 0033:0x7fdc0afbf332 [ 5540.566253] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5540.590021] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5540.597290] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5540.603356] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5540.608417] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5540.618349] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5540.621474] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5548.056378] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5548.065914] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5548.116731] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5548.124621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5548.138088] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5548.142703] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5548.166759] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5548.174229] RAX: 0000000000007e39 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5548.180897] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5548.190756] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5548.199666] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5548.205693] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5548.216223] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5548.223090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5548.230654] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5548.240935] Call Trace: [ 5548.243165] [ 5548.248784] ? show_regs.cold.9+0x22/0x2f [ 5548.253902] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5548.259573] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5548.262264] ? watchdog+0x40/0x40 [ 5548.265352] ? hrtimer_interrupt+0x128/0x2c0 [ 5548.266922] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5548.273259] ? apic_timer_interrupt+0xf/0x20 [ 5548.277819] [ 5548.280278] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5548.284965] do_raw_spin_lock+0x177/0x190 [ 5548.287887] _raw_spin_lock+0x21/0x30 [ 5548.291156] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5548.296263] ? do_raw_spin_unlock+0x75/0x190 [ 5548.303632] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5548.308665] ? do_wait_intr+0xf0/0xf0 [ 5548.312863] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5548.321949] mdc_close+0x53b/0xd70 [mdc] [ 5548.327238] lmv_close+0x18d/0x490 [lmv] [ 5548.330845] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5548.336309] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5548.342467] ll_md_close+0x274/0x800 [lustre] [ 5548.349578] ll_file_release+0x128/0xc50 [lustre] [ 5548.357456] __fput+0x10d/0x440 [ 5548.361490] ____fput+0x12/0x20 [ 5548.364517] task_work_run+0xc6/0x110 [ 5548.367910] exit_to_usermode_loop+0x1dd/0x1f0 [ 5548.377129] do_syscall_64+0x3d6/0x440 [ 5548.378563] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5548.384478] RIP: 0033:0x7fdc0afbfcd8 [ 5548.385692] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5548.407212] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5548.414953] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5548.423469] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5548.428245] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5548.434722] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5548.443513] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5552.051124] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5552.058093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5552.121838] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5552.134546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5552.138932] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5552.143752] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5552.156209] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5552.164455] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5552.175903] RAX: 0000000000006983 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5552.180616] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5552.182970] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5552.191890] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5552.195647] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5552.207661] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5552.215504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5552.222080] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5552.228621] Call Trace: [ 5552.229707] [ 5552.230886] ? show_regs.cold.9+0x22/0x2f [ 5552.236034] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5552.243666] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5552.247701] ? watchdog+0x40/0x40 [ 5552.253332] ? hrtimer_interrupt+0x128/0x2c0 [ 5552.258739] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5552.262075] ? apic_timer_interrupt+0xf/0x20 [ 5552.264213] [ 5552.265964] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5552.276894] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5552.280395] do_raw_spin_lock+0x177/0x190 [ 5552.285721] _raw_spin_lock+0x21/0x30 [ 5552.287885] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5552.291714] process_one_work+0x2c8/0x700 [ 5552.295505] worker_thread+0x296/0x6e0 [ 5552.301478] ? rescuer_thread+0x570/0x570 [ 5552.302968] kthread+0x1d1/0x200 [ 5552.303999] ? set_kthread_struct+0x70/0x70 [ 5552.308341] ret_from_fork+0x1f/0x30 [ 5564.052064] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5564.063631] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5564.149701] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5564.169480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5564.186422] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5564.199943] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5564.217871] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5564.234708] RAX: 000000000000530b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5564.244923] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5564.249980] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5564.258007] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5564.266817] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5564.269392] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5564.278426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5564.280472] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5564.289838] Call Trace: [ 5564.292988] [ 5564.294698] ? show_regs.cold.9+0x22/0x2f [ 5564.300627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5564.308947] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5564.312198] ? watchdog+0x40/0x40 [ 5564.313645] ? hrtimer_interrupt+0x128/0x2c0 [ 5564.315207] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5564.323271] ? apic_timer_interrupt+0xf/0x20 [ 5564.327512] [ 5564.334760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5564.341610] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5564.343538] ? dump_stack+0xf6/0x10e [ 5564.350640] do_raw_spin_lock+0x177/0x190 [ 5564.354541] _raw_spin_lock+0x21/0x30 [ 5564.356463] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5564.363361] obd_mod_free+0x46/0x140 [ptlrpc] [ 5564.365078] mdc_commit_open+0x91/0xb0 [mdc] [ 5564.371331] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5564.374554] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5564.387007] after_reply+0xe2c/0x1780 [ptlrpc] [ 5564.389401] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5564.393990] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5564.399944] ? do_wait_intr+0xf0/0xf0 [ 5564.402773] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5564.404545] kthread+0x1d1/0x200 [ 5564.409408] ? set_kthread_struct+0x70/0x70 [ 5564.415645] ret_from_fork+0x1f/0x30 [ 5568.059197] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5568.074936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5568.124102] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5568.141126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5568.155530] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5568.166348] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5568.179898] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5568.183095] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5568.185566] RDX: 0000000000006e76 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5568.188216] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5568.190829] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5568.193682] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5568.196693] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5568.201138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5568.203803] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5568.210741] Call Trace: [ 5568.215604] [ 5568.217945] ? show_regs.cold.9+0x22/0x2f [ 5568.223645] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5568.225180] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5568.233836] ? watchdog+0x40/0x40 [ 5568.235850] ? hrtimer_interrupt+0x128/0x2c0 [ 5568.243067] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5568.248238] ? apic_timer_interrupt+0xf/0x20 [ 5568.253738] [ 5568.254748] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5568.257530] do_raw_spin_lock+0x177/0x190 [ 5568.259602] _raw_spin_lock+0x21/0x30 [ 5568.261608] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5568.263935] ? do_raw_spin_unlock+0x75/0x190 [ 5568.265901] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5568.268691] ? do_wait_intr+0xf0/0xf0 [ 5568.270215] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5568.273679] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5568.287881] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5568.290214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5568.297745] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5568.305927] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5568.307487] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5568.311648] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5568.314840] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5568.318663] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5568.324811] lmv_intent_open+0x338/0x1150 [lmv] [ 5568.328868] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5568.332633] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5568.336640] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5568.339563] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5568.345497] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5568.347887] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5568.357811] ? do_raw_spin_unlock+0x75/0x190 [ 5568.362483] ? _raw_spin_unlock+0x12/0x30 [ 5568.365305] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5568.368655] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5568.371658] ll_atomic_open+0x36a/0xd70 [lustre] [ 5568.376915] lookup_open+0x356/0xb40 [ 5568.379019] do_last+0x313/0xfc0 [ 5568.382707] ? nd_jump_root+0xe5/0x160 [ 5568.388514] ? path_init+0x437/0x520 [ 5568.392489] path_openat+0xf7/0x500 [ 5568.397976] ? do_raw_spin_unlock+0x75/0x190 [ 5568.400661] do_filp_open+0x99/0x140 [ 5568.403761] ? getname_flags+0x6e/0x330 [ 5568.411118] ? __check_object_size+0xff/0x256 [ 5568.419053] ? do_raw_spin_unlock+0x75/0x190 [ 5568.424224] ? _raw_spin_unlock+0x12/0x30 [ 5568.425631] do_sys_openat2+0x2b4/0x410 [ 5568.426974] do_sys_open+0x73/0xa0 [ 5568.431847] __x64_sys_openat+0x24/0x30 [ 5568.433755] do_syscall_64+0xc1/0x440 [ 5568.436620] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5568.441688] RIP: 0033:0x7fdc0afbf332 [ 5568.444373] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5568.454835] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5568.458776] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5568.461815] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5568.465526] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5568.472716] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5568.480849] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5568.481370] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3543s! [ 5568.501263] Showing busy workqueues and worker pools: [ 5568.505198] workqueue events: flags=0x0 [ 5568.511892] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5568.516150] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5568.524843] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5568.533891] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5568.543793] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5568.553624] pending: kfree_rcu_monitor [ 5568.557136] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5568.560114] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5568.579083] workqueue events_long: flags=0x0 [ 5568.585269] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5568.594638] pending: lu_objects_flush [obdclass] [ 5568.598673] workqueue events_unbound: flags=0x2 [ 5568.603935] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5568.607564] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5568.616839] workqueue events_power_efficient: flags=0x80 [ 5568.621744] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5568.625696] pending: neigh_periodic_work, check_lifetime [ 5568.631820] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5568.640763] pending: do_cache_clean, neigh_periodic_work [ 5568.646246] workqueue mm_percpu_wq: flags=0x8 [ 5568.649087] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5568.653885] pending: lru_add_drain_per_cpu [ 5568.661844] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5568.666609] pending: vmstat_update, lru_add_drain_per_cpu [ 5568.669201] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5568.672648] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5568.678223] workqueue writeback: flags=0x4a [ 5568.683442] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5568.689291] pending: wb_workfn, wb_workfn [ 5568.692853] workqueue xprtiod: flags=0xa [ 5568.699544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5568.702564] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5568.706762] workqueue ipv6_addrconf: flags=0x40008 [ 5568.709254] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5568.713641] pending: addrconf_verify_work [ 5568.716935] workqueue knbd0-recv: flags=0x1a [ 5568.718650] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5568.725714] in-flight: 655:recv_work [ 5568.728405] workqueue ptlrpc_pinger: flags=0x2 [ 5568.730415] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5568.735765] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5568.740257] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3543s workers=4 idle: 36396 38 11188 [ 5568.743587] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5559s workers=2 idle: 658 [ 5576.057300] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5576.066041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5576.126382] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5576.136423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5576.146389] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5576.151518] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5576.177347] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5576.185466] RAX: 0000000000007d87 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5576.207002] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5576.215568] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5576.223889] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5576.233381] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5576.255741] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5576.258656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5576.276358] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5576.288244] Call Trace: [ 5576.289184] [ 5576.297654] ? show_regs.cold.9+0x22/0x2f [ 5576.308502] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5576.316404] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5576.334374] ? watchdog+0x40/0x40 [ 5576.342629] ? hrtimer_interrupt+0x128/0x2c0 [ 5576.344367] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5576.353347] ? apic_timer_interrupt+0xf/0x20 [ 5576.358434] [ 5576.367384] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5576.376012] do_raw_spin_lock+0x177/0x190 [ 5576.379143] _raw_spin_lock+0x21/0x30 [ 5576.380547] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5576.403427] ? do_raw_spin_unlock+0x75/0x190 [ 5576.411365] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5576.422197] ? do_wait_intr+0xf0/0xf0 [ 5576.433714] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5576.443553] mdc_close+0x53b/0xd70 [mdc] [ 5576.452443] lmv_close+0x18d/0x490 [lmv] [ 5576.454011] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5576.461018] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5576.467230] ll_md_close+0x274/0x800 [lustre] [ 5576.472173] ll_file_release+0x128/0xc50 [lustre] [ 5576.476715] __fput+0x10d/0x440 [ 5576.479801] ____fput+0x12/0x20 [ 5576.481561] task_work_run+0xc6/0x110 [ 5576.483540] exit_to_usermode_loop+0x1dd/0x1f0 [ 5576.487396] do_syscall_64+0x3d6/0x440 [ 5576.490288] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5576.503888] RIP: 0033:0x7fdc0afbfcd8 [ 5576.513584] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5576.548489] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5576.559589] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5576.564105] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5576.570235] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5576.587203] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5576.598033] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5580.051966] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5580.064935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5580.162053] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5580.179343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5580.188607] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5580.197204] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5580.203258] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5580.229663] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5580.238469] RAX: 0000000000005318 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5580.248232] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5580.258657] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5580.261128] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5580.284957] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5580.299162] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5580.315097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5580.326138] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5580.331235] Call Trace: [ 5580.334203] [ 5580.337518] ? show_regs.cold.9+0x22/0x2f [ 5580.340181] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5580.348138] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5580.359717] ? watchdog+0x40/0x40 [ 5580.361310] ? hrtimer_interrupt+0x128/0x2c0 [ 5580.368917] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5580.370717] ? apic_timer_interrupt+0xf/0x20 [ 5580.384629] [ 5580.385397] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5580.395668] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5580.409292] do_raw_spin_lock+0x177/0x190 [ 5580.410659] _raw_spin_lock+0x21/0x30 [ 5580.423036] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5580.437317] process_one_work+0x2c8/0x700 [ 5580.439230] worker_thread+0x296/0x6e0 [ 5580.444595] ? rescuer_thread+0x570/0x570 [ 5580.451195] kthread+0x1d1/0x200 [ 5580.456589] ? set_kthread_struct+0x70/0x70 [ 5580.472312] ret_from_fork+0x1f/0x30 [ 5596.048148] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5596.052208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5596.057351] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5596.091937] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5596.091952] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5596.091956] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5596.091971] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5596.091974] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5596.092162] RAX: 0000000000003d06 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5596.095287] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5596.095364] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5596.095371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5596.095374] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5596.095387] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5596.095390] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5596.095395] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5596.095397] RDX: 0000000000004e58 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5596.095400] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5596.095402] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5596.095404] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5596.095408] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5596.095410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5596.095413] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5596.095420] Call Trace: [ 5596.095425] [ 5596.095430] ? show_regs.cold.9+0x22/0x2f [ 5596.095438] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5596.095444] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5596.095448] ? watchdog+0x40/0x40 [ 5596.095452] ? hrtimer_interrupt+0x128/0x2c0 [ 5596.095455] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5596.095463] ? apic_timer_interrupt+0xf/0x20 [ 5596.095467] [ 5596.095469] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5596.095475] do_raw_spin_lock+0x177/0x190 [ 5596.095479] _raw_spin_lock+0x21/0x30 [ 5596.095484] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5596.096183] ? do_raw_spin_unlock+0x75/0x190 [ 5596.096190] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5596.096405] ? do_wait_intr+0xf0/0xf0 [ 5596.097176] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5596.097951] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5596.102207] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5596.111048] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5596.111061] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5596.111064] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5596.111066] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5596.111070] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5596.111076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5596.111078] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5596.111085] Call Trace: [ 5596.111092] [ 5596.111098] ? show_regs.cold.9+0x22/0x2f [ 5596.111116] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5596.111124] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5596.111129] ? watchdog+0x40/0x40 [ 5596.111133] ? hrtimer_interrupt+0x128/0x2c0 [ 5596.111138] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5596.111150] ? apic_timer_interrupt+0xf/0x20 [ 5596.111158] [ 5596.111161] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5596.111169] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5596.111173] ? dump_stack+0xf6/0x10e [ 5596.111181] do_raw_spin_lock+0x177/0x190 [ 5596.111186] _raw_spin_lock+0x21/0x30 [ 5596.111196] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5596.113893] obd_mod_free+0x46/0x140 [ptlrpc] [ 5596.114127] mdc_commit_open+0x91/0xb0 [mdc] [ 5596.114163] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5596.114378] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5596.115450] after_reply+0xe2c/0x1780 [ptlrpc] [ 5596.116282] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5596.121346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5596.138594] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5596.138863] ? do_wait_intr+0xf0/0xf0 [ 5596.138883] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5596.139093] kthread+0x1d1/0x200 [ 5596.139099] ? set_kthread_struct+0x70/0x70 [ 5596.139106] ret_from_fork+0x1f/0x30 [ 5596.429182] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5596.432779] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5596.434261] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5596.436768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5596.440118] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5596.441812] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5596.444292] lmv_intent_open+0x338/0x1150 [lmv] [ 5596.447767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5596.454758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5596.463038] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5596.468094] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5596.469384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5596.476064] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5596.482104] ? do_raw_spin_unlock+0x75/0x190 [ 5596.485969] ? _raw_spin_unlock+0x12/0x30 [ 5596.489371] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5596.493060] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5596.495347] ll_atomic_open+0x36a/0xd70 [lustre] [ 5596.497280] lookup_open+0x356/0xb40 [ 5596.500151] do_last+0x313/0xfc0 [ 5596.500990] ? nd_jump_root+0xe5/0x160 [ 5596.506633] ? path_init+0x437/0x520 [ 5596.508250] path_openat+0xf7/0x500 [ 5596.511033] ? do_raw_spin_unlock+0x75/0x190 [ 5596.513170] do_filp_open+0x99/0x140 [ 5596.519623] ? getname_flags+0x6e/0x330 [ 5596.521096] ? __check_object_size+0xff/0x256 [ 5596.523933] ? do_raw_spin_unlock+0x75/0x190 [ 5596.526174] ? _raw_spin_unlock+0x12/0x30 [ 5596.528660] do_sys_openat2+0x2b4/0x410 [ 5596.530228] do_sys_open+0x73/0xa0 [ 5596.536315] __x64_sys_openat+0x24/0x30 [ 5596.539705] do_syscall_64+0xc1/0x440 [ 5596.541302] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5596.546308] RIP: 0033:0x7fdc0afbf332 [ 5596.548237] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5596.560214] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5596.567022] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5596.569849] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5596.572509] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5596.575046] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5596.578267] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5599.199677] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3574s! [ 5599.204423] Showing busy workqueues and worker pools: [ 5599.207161] workqueue events: flags=0x0 [ 5599.208876] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5599.214462] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5599.218223] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5599.228575] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5599.234672] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5599.237568] pending: kfree_rcu_monitor [ 5599.239390] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5599.246134] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5599.255522] workqueue events_long: flags=0x0 [ 5599.259760] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5599.269667] pending: lu_objects_flush [obdclass] [ 5599.274251] workqueue events_unbound: flags=0x2 [ 5599.276558] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5599.280334] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5599.287130] workqueue events_power_efficient: flags=0x80 [ 5599.292672] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5599.299494] pending: neigh_periodic_work, check_lifetime [ 5599.303249] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5599.306702] pending: do_cache_clean, neigh_periodic_work [ 5599.310186] workqueue mm_percpu_wq: flags=0x8 [ 5599.313902] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5599.318035] pending: lru_add_drain_per_cpu [ 5599.321612] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5599.326276] pending: vmstat_update, lru_add_drain_per_cpu [ 5599.329180] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5599.334183] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5599.336866] workqueue writeback: flags=0x4a [ 5599.340207] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5599.343469] pending: wb_workfn, wb_workfn [ 5599.345924] workqueue xprtiod: flags=0xa [ 5599.347184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5599.348849] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5599.352710] workqueue ipv6_addrconf: flags=0x40008 [ 5599.354011] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5599.356387] pending: addrconf_verify_work [ 5599.358718] workqueue knbd0-recv: flags=0x1a [ 5599.361213] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5599.366905] in-flight: 655:recv_work [ 5599.368915] workqueue ptlrpc_pinger: flags=0x2 [ 5599.371179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5599.374124] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5599.377828] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3574s workers=4 idle: 36396 38 11188 [ 5599.382718] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5589s workers=2 idle: 658 [ 5604.060503] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5604.069977] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5604.113857] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5604.119372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5604.123390] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5604.126396] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5604.134977] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5604.138306] RAX: 0000000000003c14 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5604.141186] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5604.143886] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5604.149260] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5604.154267] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5604.164724] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5604.170932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5604.176096] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5604.184622] Call Trace: [ 5604.185893] [ 5604.187715] ? show_regs.cold.9+0x22/0x2f [ 5604.192086] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5604.198711] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5604.202520] ? watchdog+0x40/0x40 [ 5604.205532] ? hrtimer_interrupt+0x128/0x2c0 [ 5604.210219] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5604.213921] ? apic_timer_interrupt+0xf/0x20 [ 5604.217512] [ 5604.218447] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5604.223960] do_raw_spin_lock+0x177/0x190 [ 5604.227988] _raw_spin_lock+0x21/0x30 [ 5604.232291] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5604.240123] ? do_raw_spin_unlock+0x75/0x190 [ 5604.243294] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5604.255113] ? do_wait_intr+0xf0/0xf0 [ 5604.256811] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5604.260325] mdc_close+0x53b/0xd70 [mdc] [ 5604.266111] lmv_close+0x18d/0x490 [lmv] [ 5604.268069] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5604.272320] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5604.274178] ll_md_close+0x274/0x800 [lustre] [ 5604.279017] ll_file_release+0x128/0xc50 [lustre] [ 5604.283234] __fput+0x10d/0x440 [ 5604.284433] ____fput+0x12/0x20 [ 5604.285571] task_work_run+0xc6/0x110 [ 5604.291665] exit_to_usermode_loop+0x1dd/0x1f0 [ 5604.298050] do_syscall_64+0x3d6/0x440 [ 5604.299288] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5604.303278] RIP: 0033:0x7fdc0afbfcd8 [ 5604.306996] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5604.323117] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5604.328487] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5604.332188] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5604.335313] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5604.338685] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5604.341220] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5608.056481] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5608.066009] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5608.115296] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5608.128628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5608.133202] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5608.150999] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5608.153267] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5608.182684] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5608.186759] RAX: 0000000000006383 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5608.191780] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5608.198967] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5608.207987] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5608.210487] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5608.215654] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5608.228139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5608.234655] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5608.238148] Call Trace: [ 5608.239939] [ 5608.240935] ? show_regs.cold.9+0x22/0x2f [ 5608.243146] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5608.245643] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5608.247595] ? watchdog+0x40/0x40 [ 5608.252929] ? hrtimer_interrupt+0x128/0x2c0 [ 5608.255018] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5608.260806] ? apic_timer_interrupt+0xf/0x20 [ 5608.262216] [ 5608.263004] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5608.271015] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5608.275203] do_raw_spin_lock+0x177/0x190 [ 5608.280722] _raw_spin_lock+0x21/0x30 [ 5608.286868] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5608.292298] process_one_work+0x2c8/0x700 [ 5608.293750] worker_thread+0x296/0x6e0 [ 5608.295072] ? rescuer_thread+0x570/0x570 [ 5608.301023] kthread+0x1d1/0x200 [ 5608.302758] ? set_kthread_struct+0x70/0x70 [ 5608.305379] ret_from_fork+0x1f/0x30 [ 5624.048841] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5624.056959] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5624.059770] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [ 5624.063930] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5624.064767] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5624.064775] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5624.064779] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5624.064793] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5624.064798] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5624.064803] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5624.064805] RDX: 000000000000126b RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5624.064808] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5624.064810] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5624.064812] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5624.064816] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5624.064819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5624.064821] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5624.064831] Call Trace: [ 5624.064836] [ 5624.064842] ? show_regs.cold.9+0x22/0x2f [ 5624.064852] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5624.064858] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5624.064863] ? watchdog+0x40/0x40 [ 5624.064867] ? hrtimer_interrupt+0x128/0x2c0 [ 5624.064871] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5624.064879] ? apic_timer_interrupt+0xf/0x20 [ 5624.064884] [ 5624.064887] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5624.064892] do_raw_spin_lock+0x177/0x190 [ 5624.064896] _raw_spin_lock+0x21/0x30 [ 5624.064903] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5624.065829] ? do_raw_spin_unlock+0x75/0x190 [ 5624.065836] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5624.066439] ? do_wait_intr+0xf0/0xf0 [ 5624.066450] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5624.066650] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5624.085663] lfsck(O) [ 5624.121640] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5624.126800] obdecho(O) [ 5624.130296] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5624.132217] mgc(O) [ 5624.140687] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5624.140741] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5624.140771] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5624.140799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5624.140919] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5624.141360] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5624.141406] lmv_intent_open+0x338/0x1150 [lmv] [ 5624.141436] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5624.141522] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5624.141601] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5624.141623] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5624.141706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5624.141788] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5624.141872] ? do_raw_spin_unlock+0x75/0x190 [ 5624.141880] ? _raw_spin_unlock+0x12/0x30 [ 5624.141887] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5624.141916] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5624.141940] ll_atomic_open+0x36a/0xd70 [lustre] [ 5624.142213] lookup_open+0x356/0xb40 [ 5624.142225] do_last+0x313/0xfc0 [ 5624.142229] ? nd_jump_root+0xe5/0x160 [ 5624.142235] ? path_init+0x437/0x520 [ 5624.142239] path_openat+0xf7/0x500 [ 5624.142244] ? do_raw_spin_unlock+0x75/0x190 [ 5624.142249] do_filp_open+0x99/0x140 [ 5624.142255] ? getname_flags+0x6e/0x330 [ 5624.142260] ? __check_object_size+0xff/0x256 [ 5624.142267] ? do_raw_spin_unlock+0x75/0x190 [ 5624.142272] ? _raw_spin_unlock+0x12/0x30 [ 5624.142277] do_sys_openat2+0x2b4/0x410 [ 5624.142282] do_sys_open+0x73/0xa0 [ 5624.142287] __x64_sys_openat+0x24/0x30 [ 5624.142291] do_syscall_64+0xc1/0x440 [ 5624.142312] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5624.142320] RIP: 0033:0x7fdc0afbf332 [ 5624.142329] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5624.142333] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5624.142338] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5624.142340] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5624.142343] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5624.142345] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5624.142347] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5624.522569] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5624.539502] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5624.546727] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5624.553361] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5624.555542] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5624.567426] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5624.571333] RAX: 0000000000004c86 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5624.574492] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5624.577757] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5624.581425] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5624.584960] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5624.589739] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5624.593074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5624.595622] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5624.598827] Call Trace: [ 5624.600382] [ 5624.601096] ? show_regs.cold.9+0x22/0x2f [ 5624.603202] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5624.605674] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5624.609679] ? watchdog+0x40/0x40 [ 5624.611669] ? hrtimer_interrupt+0x128/0x2c0 [ 5624.614030] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5624.619063] ? apic_timer_interrupt+0xf/0x20 [ 5624.622445] [ 5624.623613] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5624.626879] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5624.632656] ? dump_stack+0xf6/0x10e [ 5624.634696] do_raw_spin_lock+0x177/0x190 [ 5624.637808] _raw_spin_lock+0x21/0x30 [ 5624.639004] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5624.646906] obd_mod_free+0x46/0x140 [ptlrpc] [ 5624.650182] mdc_commit_open+0x91/0xb0 [mdc] [ 5624.653463] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5624.668874] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5624.671044] after_reply+0xe2c/0x1780 [ptlrpc] [ 5624.675685] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5624.678830] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5624.681578] ? do_wait_intr+0xf0/0xf0 [ 5624.682944] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5624.689686] kthread+0x1d1/0x200 [ 5624.694426] ? set_kthread_struct+0x70/0x70 [ 5624.696006] ret_from_fork+0x1f/0x30 [ 5629.407130] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3604s! [ 5629.412624] Showing busy workqueues and worker pools: [ 5629.415588] workqueue events: flags=0x0 [ 5629.417320] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5629.420474] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5629.423773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5629.427137] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5629.432626] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5629.435954] pending: kfree_rcu_monitor [ 5629.437857] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5629.441468] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5629.447793] workqueue events_long: flags=0x0 [ 5629.449681] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5629.452937] pending: lu_objects_flush [obdclass] [ 5629.455563] workqueue events_unbound: flags=0x2 [ 5629.457582] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5629.460308] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5629.465339] workqueue events_power_efficient: flags=0x80 [ 5629.467626] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5629.470724] pending: neigh_periodic_work, check_lifetime [ 5629.473399] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5629.476639] pending: do_cache_clean, neigh_periodic_work [ 5629.479484] workqueue mm_percpu_wq: flags=0x8 [ 5629.481270] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5629.484249] pending: lru_add_drain_per_cpu [ 5629.486450] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5629.489720] pending: vmstat_update, lru_add_drain_per_cpu [ 5629.492247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5629.495346] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5629.498282] workqueue writeback: flags=0x4a [ 5629.500135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5629.503044] pending: wb_workfn, wb_workfn [ 5629.505159] workqueue xprtiod: flags=0xa [ 5629.507312] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5629.510674] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5629.519791] workqueue ipv6_addrconf: flags=0x40008 [ 5629.522681] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5629.527143] pending: addrconf_verify_work [ 5629.529042] workqueue knbd0-recv: flags=0x1a [ 5629.530525] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5629.533600] in-flight: 655:recv_work [ 5629.537576] workqueue ptlrpc_pinger: flags=0x2 [ 5629.540731] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5629.547276] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5629.551110] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3604s workers=4 idle: 36396 38 11188 [ 5629.554633] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5620s workers=2 idle: 658 [ 5632.058493] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5632.065308] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5632.092056] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5632.121110] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5632.132576] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5632.142489] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5632.165191] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5632.179947] RAX: 000000000000138a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5632.187826] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5632.192879] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5632.194944] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5632.197441] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5632.200007] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5632.202650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5632.204943] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5632.207539] Call Trace: [ 5632.208439] [ 5632.209193] ? show_regs.cold.9+0x22/0x2f [ 5632.210667] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5632.212328] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5632.213983] ? watchdog+0x40/0x40 [ 5632.215168] ? hrtimer_interrupt+0x128/0x2c0 [ 5632.216625] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5632.218358] ? apic_timer_interrupt+0xf/0x20 [ 5632.219821] [ 5632.220535] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5632.222539] do_raw_spin_lock+0x177/0x190 [ 5632.223983] _raw_spin_lock+0x21/0x30 [ 5632.225268] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5632.227381] ? do_raw_spin_unlock+0x75/0x190 [ 5632.228931] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5632.231029] ? do_wait_intr+0xf0/0xf0 [ 5632.232404] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5632.234653] mdc_close+0x53b/0xd70 [mdc] [ 5632.236194] lmv_close+0x18d/0x490 [lmv] [ 5632.237683] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5632.239817] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5632.241677] ll_md_close+0x274/0x800 [lustre] [ 5632.243470] ll_file_release+0x128/0xc50 [lustre] [ 5632.245294] __fput+0x10d/0x440 [ 5632.246419] ____fput+0x12/0x20 [ 5632.247519] task_work_run+0xc6/0x110 [ 5632.248793] exit_to_usermode_loop+0x1dd/0x1f0 [ 5632.250455] do_syscall_64+0x3d6/0x440 [ 5632.251936] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5632.253677] RIP: 0033:0x7fdc0afbfcd8 [ 5632.255257] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5632.262126] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5632.264797] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5632.277617] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5632.290725] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5632.295449] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5632.300555] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5636.051118] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5636.058010] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5636.079018] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5636.083989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5636.087837] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5636.090993] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5636.093978] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5636.101996] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5636.105118] RAX: 0000000000007a14 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5636.108311] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5636.111468] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5636.114490] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5636.117916] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5636.121908] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5636.125186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5636.127919] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5636.130635] Call Trace: [ 5636.131647] [ 5636.132658] ? show_regs.cold.9+0x22/0x2f [ 5636.134144] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5636.136199] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5636.138065] ? watchdog+0x40/0x40 [ 5636.139460] ? hrtimer_interrupt+0x128/0x2c0 [ 5636.141202] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5636.142928] ? apic_timer_interrupt+0xf/0x20 [ 5636.144730] [ 5636.145653] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5636.147844] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5636.149990] do_raw_spin_lock+0x177/0x190 [ 5636.151884] _raw_spin_lock+0x21/0x30 [ 5636.153275] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5636.155685] process_one_work+0x2c8/0x700 [ 5636.157506] worker_thread+0x296/0x6e0 [ 5636.158928] ? rescuer_thread+0x570/0x570 [ 5636.160936] kthread+0x1d1/0x200 [ 5636.162306] ? set_kthread_struct+0x70/0x70 [ 5636.164045] ret_from_fork+0x1f/0x30 [ 5652.052073] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5652.056384] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5652.091718] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5652.096694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5652.100388] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5652.102386] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5652.111021] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5652.113592] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5652.116006] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5652.118565] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5652.121235] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5652.124060] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5652.127469] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5652.130385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5652.132449] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5652.135593] Call Trace: [ 5652.136563] [ 5652.137465] ? show_regs.cold.9+0x22/0x2f [ 5652.139107] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5652.141016] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5652.142585] ? watchdog+0x40/0x40 [ 5652.144082] ? hrtimer_interrupt+0x128/0x2c0 [ 5652.145803] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5652.147830] ? apic_timer_interrupt+0xf/0x20 [ 5652.149176] [ 5652.149845] ? native_safe_halt+0x2e/0x50 [ 5652.151364] kvm_wait+0xcd/0xe0 [ 5652.152703] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5652.154306] do_raw_spin_lock+0x177/0x190 [ 5652.155951] _raw_spin_lock+0x21/0x30 [ 5652.157345] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5652.159680] ? do_raw_spin_unlock+0x75/0x190 [ 5652.161379] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5652.163720] ? do_wait_intr+0xf0/0xf0 [ 5652.165120] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5652.167466] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5652.169812] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5652.172346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5652.174772] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5652.177272] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5652.178996] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5652.180583] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5652.182987] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5652.185969] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5652.188136] lmv_intent_open+0x338/0x1150 [lmv] [ 5652.190165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5652.192109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5652.194155] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5652.195923] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5652.197737] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5652.200330] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5652.202442] ? do_raw_spin_unlock+0x75/0x190 [ 5652.204143] ? _raw_spin_unlock+0x12/0x30 [ 5652.205767] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5652.207920] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5652.210218] ll_atomic_open+0x36a/0xd70 [lustre] [ 5652.211953] lookup_open+0x356/0xb40 [ 5652.213334] do_last+0x313/0xfc0 [ 5652.215220] ? nd_jump_root+0xe5/0x160 [ 5652.216911] ? path_init+0x437/0x520 [ 5652.218224] path_openat+0xf7/0x500 [ 5652.219715] ? do_raw_spin_unlock+0x75/0x190 [ 5652.221241] do_filp_open+0x99/0x140 [ 5652.222666] ? getname_flags+0x6e/0x330 [ 5652.224069] ? __check_object_size+0xff/0x256 [ 5652.226877] ? do_raw_spin_unlock+0x75/0x190 [ 5652.229607] ? _raw_spin_unlock+0x12/0x30 [ 5652.231189] do_sys_openat2+0x2b4/0x410 [ 5652.232629] do_sys_open+0x73/0xa0 [ 5652.233925] __x64_sys_openat+0x24/0x30 [ 5652.235269] do_syscall_64+0xc1/0x440 [ 5652.236731] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5652.238584] RIP: 0033:0x7fdc0afbf332 [ 5652.239969] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5652.246688] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5652.249535] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5652.251970] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5652.254478] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5652.257047] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5652.259113] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5656.048032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5656.050602] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5656.077360] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5656.089867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5656.094743] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5656.096721] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5656.111681] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5656.119980] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5656.123215] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5656.125230] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5656.132295] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5656.136072] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5656.139380] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5656.143479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5656.146734] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5656.149626] Call Trace: [ 5656.150325] [ 5656.150957] ? show_regs.cold.9+0x22/0x2f [ 5656.152146] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5656.153480] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5656.154677] ? watchdog+0x40/0x40 [ 5656.156096] ? hrtimer_interrupt+0x128/0x2c0 [ 5656.157258] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5656.158525] ? apic_timer_interrupt+0xf/0x20 [ 5656.160084] [ 5656.160613] ? native_safe_halt+0x2e/0x50 [ 5656.161655] kvm_wait+0xcd/0xe0 [ 5656.162852] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5656.164330] ? dump_stack+0xf6/0x10e [ 5656.165303] do_raw_spin_lock+0x177/0x190 [ 5656.166331] _raw_spin_lock+0x21/0x30 [ 5656.167487] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5656.169583] obd_mod_free+0x46/0x140 [ptlrpc] [ 5656.171159] mdc_commit_open+0x91/0xb0 [mdc] [ 5656.172469] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5656.174252] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5656.176718] after_reply+0xe2c/0x1780 [ptlrpc] [ 5656.182987] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5656.187120] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5656.195209] ? do_wait_intr+0xf0/0xf0 [ 5656.196956] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5656.202134] kthread+0x1d1/0x200 [ 5656.203736] ? set_kthread_struct+0x70/0x70 [ 5656.205287] ret_from_fork+0x1f/0x30 [ 5659.615103] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3634s! [ 5659.619049] Showing busy workqueues and worker pools: [ 5659.620877] workqueue events: flags=0x0 [ 5659.622218] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5659.624874] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5659.627354] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5659.629967] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5659.634523] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5659.637484] pending: kfree_rcu_monitor [ 5659.639021] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5659.642162] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5659.647072] workqueue events_long: flags=0x0 [ 5659.648603] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5659.651474] pending: lu_objects_flush [obdclass] [ 5659.653721] workqueue events_unbound: flags=0x2 [ 5659.655554] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5659.658129] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5659.662573] workqueue events_power_efficient: flags=0x80 [ 5659.664863] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5659.667600] pending: neigh_periodic_work, check_lifetime [ 5659.669849] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5659.672424] pending: do_cache_clean, neigh_periodic_work [ 5659.674301] workqueue mm_percpu_wq: flags=0x8 [ 5659.675907] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5659.678698] pending: lru_add_drain_per_cpu [ 5659.680664] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5659.683095] pending: vmstat_update, lru_add_drain_per_cpu [ 5659.684715] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5659.686370] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5659.688835] workqueue writeback: flags=0x4a [ 5659.690321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5659.693121] pending: wb_workfn, wb_workfn [ 5659.695051] workqueue xprtiod: flags=0xa [ 5659.696728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5659.699296] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5659.701789] workqueue ipv6_addrconf: flags=0x40008 [ 5659.703062] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5659.704738] pending: addrconf_verify_work [ 5659.706165] workqueue knbd0-recv: flags=0x1a [ 5659.707427] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5659.710040] in-flight: 655:recv_work [ 5659.711721] workqueue ptlrpc_pinger: flags=0x2 [ 5659.713506] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5659.716058] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5659.718496] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3634s workers=4 idle: 36396 38 11188 [ 5659.721974] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5650s workers=2 idle: 658 [ 5660.054253] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5660.057562] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5660.106163] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5660.117615] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5660.127551] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5660.131988] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5660.145500] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5660.156301] RAX: 0000000000001909 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5660.163972] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5660.172324] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5660.182244] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5660.185729] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5660.198959] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5660.209587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5660.212722] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5660.215314] Call Trace: [ 5660.216529] [ 5660.219773] ? show_regs.cold.9+0x22/0x2f [ 5660.222384] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5660.224637] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5660.228613] ? watchdog+0x40/0x40 [ 5660.230592] ? hrtimer_interrupt+0x128/0x2c0 [ 5660.232705] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5660.240168] ? apic_timer_interrupt+0xf/0x20 [ 5660.242108] [ 5660.243255] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5660.245623] do_raw_spin_lock+0x177/0x190 [ 5660.251302] _raw_spin_lock+0x21/0x30 [ 5660.253415] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5660.257049] ? do_raw_spin_unlock+0x75/0x190 [ 5660.260778] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5660.264920] ? do_wait_intr+0xf0/0xf0 [ 5660.266828] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5660.271449] mdc_close+0x53b/0xd70 [mdc] [ 5660.274633] lmv_close+0x18d/0x490 [lmv] [ 5660.276653] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5660.280166] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5660.283983] ll_md_close+0x274/0x800 [lustre] [ 5660.287298] ll_file_release+0x128/0xc50 [lustre] [ 5660.293621] __fput+0x10d/0x440 [ 5660.295197] ____fput+0x12/0x20 [ 5660.298139] task_work_run+0xc6/0x110 [ 5660.300975] exit_to_usermode_loop+0x1dd/0x1f0 [ 5660.304836] do_syscall_64+0x3d6/0x440 [ 5660.306962] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5660.311297] RIP: 0033:0x7fdc0afbfcd8 [ 5660.315663] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5660.326970] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5660.330612] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5660.334190] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5660.341233] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5660.345931] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5660.348540] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5664.059938] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 5664.069164] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5664.103237] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5664.111574] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5664.115895] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5664.118733] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5664.121571] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5664.127267] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5664.129630] RAX: 0000000000007923 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5664.132979] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5664.136601] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5664.140117] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5664.142765] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5664.145863] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5664.149848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5664.154415] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5664.158214] Call Trace: [ 5664.159379] [ 5664.160710] ? show_regs.cold.9+0x22/0x2f [ 5664.163623] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5664.169166] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5664.173635] ? watchdog+0x40/0x40 [ 5664.177542] ? hrtimer_interrupt+0x128/0x2c0 [ 5664.179944] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5664.183499] ? apic_timer_interrupt+0xf/0x20 [ 5664.186782] [ 5664.191895] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5664.195951] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5664.200275] do_raw_spin_lock+0x177/0x190 [ 5664.204173] _raw_spin_lock+0x21/0x30 [ 5664.206180] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5664.210010] process_one_work+0x2c8/0x700 [ 5664.212963] worker_thread+0x296/0x6e0 [ 5664.215841] ? rescuer_thread+0x570/0x570 [ 5664.219135] kthread+0x1d1/0x200 [ 5664.224995] ? set_kthread_struct+0x70/0x70 [ 5664.227864] ret_from_fork+0x1f/0x30 [ 5679.427160] rcu: INFO: rcu_sched self-detected stall on CPU [ 5679.429969] rcu: 0-...!: (2162901 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 5679.435160] (t=3660084 jiffies g=270761 q=3551) [ 5679.439015] rcu: rcu_sched kthread starved for 3655016 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 5679.444024] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5679.448595] rcu: RCU grace-period kthread stack dump: [ 5679.451090] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5679.455496] Call Trace: [ 5679.457059] __schedule+0x351/0xcb0 [ 5679.461213] schedule+0xc0/0x180 [ 5679.462666] schedule_timeout+0xb4/0x190 [ 5679.468484] ? do_raw_spin_unlock+0x75/0x190 [ 5679.470381] ? __next_timer_interrupt+0x160/0x160 [ 5679.476753] ? prepare_to_swait_event+0xf3/0x140 [ 5679.478923] rcu_gp_kthread+0x8dd/0x15b0 [ 5679.484845] ? force_qs_rnp+0x2c0/0x2c0 [ 5679.488816] kthread+0x1d1/0x200 [ 5679.492504] ? set_kthread_struct+0x70/0x70 [ 5679.495842] ret_from_fork+0x1f/0x30 [ 5679.500113] rcu: Stack dump where RCU GP kthread last ran: [ 5679.506744] NMI backtrace for cpu 0 [ 5679.508208] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5679.521799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5679.529584] Call Trace: [ 5679.530805] [ 5679.533978] ? dump_stack+0xbb/0x10e [ 5679.538057] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5679.547512] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5679.549334] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5679.555611] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5679.562621] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5679.566721] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 5679.571408] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 5679.576877] ? update_process_times+0x6d/0xc0 [ 5679.579792] ? tick_sched_handle.isra.18+0x27/0x90 [ 5679.583207] ? tick_sched_timer+0x53/0xb0 [ 5679.587025] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5679.593036] ? tick_sched_handle.isra.18+0x90/0x90 [ 5679.597604] ? hrtimer_interrupt+0x128/0x2c0 [ 5679.599577] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5679.602960] ? apic_timer_interrupt+0xf/0x20 [ 5679.607230] [ 5679.609934] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5679.614849] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5679.619466] ? dump_stack+0xf6/0x10e [ 5679.622873] ? do_raw_spin_lock+0x177/0x190 [ 5679.628277] ? _raw_spin_lock+0x21/0x30 [ 5679.633338] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5679.636860] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5679.643447] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5679.645963] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5679.651161] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5679.657048] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5679.659196] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5679.665199] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5679.670855] ? do_wait_intr+0xf0/0xf0 [ 5679.672505] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5679.678177] ? kthread+0x1d1/0x200 [ 5679.679486] ? set_kthread_struct+0x70/0x70 [ 5679.684231] ? ret_from_fork+0x1f/0x30 [ 5679.685794] NMI backtrace for cpu 0 [ 5679.690942] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5679.697886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5679.704485] Call Trace: [ 5679.705428] [ 5679.706247] ? dump_stack+0xbb/0x10e [ 5679.710927] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5679.713900] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5679.718200] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5679.723523] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5679.725649] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5679.729804] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5679.731665] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5679.733697] ? update_process_times+0x6d/0xc0 [ 5679.735387] ? tick_sched_handle.isra.18+0x27/0x90 [ 5679.737601] ? tick_sched_timer+0x53/0xb0 [ 5679.739509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5679.741096] ? tick_sched_handle.isra.18+0x90/0x90 [ 5679.743189] ? hrtimer_interrupt+0x128/0x2c0 [ 5679.745082] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5679.746924] ? apic_timer_interrupt+0xf/0x20 [ 5679.748771] [ 5679.749592] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5679.752608] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5679.755361] ? dump_stack+0xf6/0x10e [ 5679.756783] ? do_raw_spin_lock+0x177/0x190 [ 5679.758382] ? _raw_spin_lock+0x21/0x30 [ 5679.760635] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5679.763459] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5679.765400] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5679.766869] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5679.769037] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5679.771533] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5679.773542] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5679.775730] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5679.777455] ? do_wait_intr+0xf0/0xf0 [ 5679.778552] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5679.780677] ? kthread+0x1d1/0x200 [ 5679.781824] ? set_kthread_struct+0x70/0x70 [ 5679.783280] ? ret_from_fork+0x1f/0x30 [ 5680.061600] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5680.065814] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5680.086532] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5680.090617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5680.093562] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5680.095669] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5680.101991] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5680.105688] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5680.108381] RDX: 0000000000007deb RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5680.110863] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5680.115148] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5680.119139] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5680.122749] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5680.127528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5680.129588] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5680.133683] Call Trace: [ 5680.137600] [ 5680.138187] ? show_regs.cold.9+0x22/0x2f [ 5680.139737] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5680.141822] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5680.143954] ? watchdog+0x40/0x40 [ 5680.145811] ? hrtimer_interrupt+0x128/0x2c0 [ 5680.149557] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5680.152603] ? apic_timer_interrupt+0xf/0x20 [ 5680.154542] [ 5680.155667] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5680.158097] do_raw_spin_lock+0x177/0x190 [ 5680.160103] _raw_spin_lock+0x21/0x30 [ 5680.162371] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5680.167181] ? do_raw_spin_unlock+0x75/0x190 [ 5680.169016] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5680.175063] ? do_wait_intr+0xf0/0xf0 [ 5680.176340] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5680.179983] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5680.182631] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5680.185736] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5680.188539] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5680.190968] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5680.194984] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5680.197495] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5680.204009] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5680.211294] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5680.214841] lmv_intent_open+0x338/0x1150 [lmv] [ 5680.217667] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5680.220546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5680.224252] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5680.228575] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5680.231935] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5680.234088] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5680.235990] ? do_raw_spin_unlock+0x75/0x190 [ 5680.237970] ? _raw_spin_unlock+0x12/0x30 [ 5680.239368] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5680.241735] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5680.245071] ll_atomic_open+0x36a/0xd70 [lustre] [ 5680.246930] lookup_open+0x356/0xb40 [ 5680.248523] do_last+0x313/0xfc0 [ 5680.249903] ? nd_jump_root+0xe5/0x160 [ 5680.251748] ? path_init+0x437/0x520 [ 5680.254062] path_openat+0xf7/0x500 [ 5680.255664] ? do_raw_spin_unlock+0x75/0x190 [ 5680.257264] do_filp_open+0x99/0x140 [ 5680.258742] ? getname_flags+0x6e/0x330 [ 5680.260701] ? __check_object_size+0xff/0x256 [ 5680.262894] ? do_raw_spin_unlock+0x75/0x190 [ 5680.264959] ? _raw_spin_unlock+0x12/0x30 [ 5680.266681] do_sys_openat2+0x2b4/0x410 [ 5680.268584] do_sys_open+0x73/0xa0 [ 5680.270679] __x64_sys_openat+0x24/0x30 [ 5680.272529] do_syscall_64+0xc1/0x440 [ 5680.274464] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5680.276309] RIP: 0033:0x7fdc0afbf332 [ 5680.278027] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5680.287107] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5680.291585] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5680.295783] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5680.299912] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5680.305436] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5680.309711] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5688.055710] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5688.061482] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5688.127187] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5688.137796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5688.165426] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5688.180489] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5688.197026] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5688.212324] RAX: 0000000000006d07 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5688.231040] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5688.253564] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5688.256106] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5688.275794] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5688.289142] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5688.316169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5688.325272] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5688.332019] Call Trace: [ 5688.332991] [ 5688.345719] ? show_regs.cold.9+0x22/0x2f [ 5688.365200] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5688.376843] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5688.386722] ? watchdog+0x40/0x40 [ 5688.387993] ? hrtimer_interrupt+0x128/0x2c0 [ 5688.405242] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5688.410180] ? apic_timer_interrupt+0xf/0x20 [ 5688.418142] [ 5688.421331] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5688.426344] do_raw_spin_lock+0x177/0x190 [ 5688.446698] _raw_spin_lock+0x21/0x30 [ 5688.454517] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5688.471731] ? do_raw_spin_unlock+0x75/0x190 [ 5688.488862] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5688.509761] ? do_wait_intr+0xf0/0xf0 [ 5688.521720] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5688.532838] mdc_close+0x53b/0xd70 [mdc] [ 5688.552494] lmv_close+0x18d/0x490 [lmv] [ 5688.554001] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5688.578274] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5688.586372] ll_md_close+0x274/0x800 [lustre] [ 5688.588315] ll_file_release+0x128/0xc50 [lustre] [ 5688.605730] __fput+0x10d/0x440 [ 5688.606894] ____fput+0x12/0x20 [ 5688.624655] task_work_run+0xc6/0x110 [ 5688.628762] exit_to_usermode_loop+0x1dd/0x1f0 [ 5688.636476] do_syscall_64+0x3d6/0x440 [ 5688.640522] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5688.649747] RIP: 0033:0x7fdc0afbfcd8 [ 5688.668047] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5688.713109] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5688.738200] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5688.757128] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5688.781943] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5688.794706] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5688.797192] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5689.823068] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3664s! [ 5689.839630] Showing busy workqueues and worker pools: [ 5689.856296] workqueue events: flags=0x0 [ 5689.870294] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5689.895029] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5689.918036] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5689.932514] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5689.961914] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5689.976088] pending: kfree_rcu_monitor [ 5689.977652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5689.986311] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5690.005595] workqueue events_long: flags=0x0 [ 5690.019021] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5690.032355] pending: lu_objects_flush [obdclass] [ 5690.047123] workqueue events_unbound: flags=0x2 [ 5690.054923] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5690.062585] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5690.080445] workqueue events_power_efficient: flags=0x80 [ 5690.086367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5690.093462] pending: neigh_periodic_work, check_lifetime [ 5690.097728] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5690.102395] pending: do_cache_clean, neigh_periodic_work [ 5690.104883] workqueue mm_percpu_wq: flags=0x8 [ 5690.107604] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5690.113683] pending: lru_add_drain_per_cpu [ 5690.117422] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5690.121412] pending: vmstat_update, lru_add_drain_per_cpu [ 5690.127209] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5690.133602] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5690.137305] workqueue writeback: flags=0x4a [ 5690.141370] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5690.145702] pending: wb_workfn, wb_workfn [ 5690.147801] workqueue xprtiod: flags=0xa [ 5690.150833] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5690.152881] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5690.159306] workqueue ipv6_addrconf: flags=0x40008 [ 5690.163321] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5690.167147] pending: addrconf_verify_work [ 5690.170913] workqueue knbd0-recv: flags=0x1a [ 5690.173182] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5690.177884] in-flight: 655:recv_work [ 5690.180344] workqueue ptlrpc_pinger: flags=0x2 [ 5690.182733] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5690.185993] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5690.191786] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3665s workers=4 idle: 36396 38 11188 [ 5690.196987] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5680s workers=2 idle: 658 [ 5692.051437] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 5692.070942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5692.160109] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5692.177493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5692.193995] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5692.202414] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5692.223413] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5692.244873] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5692.256044] RAX: 0000000000007f91 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5692.280232] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5692.298037] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5692.309380] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5692.319075] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5692.330437] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5692.339822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5692.362759] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5692.364832] Call Trace: [ 5692.372155] [ 5692.373256] ? show_regs.cold.9+0x22/0x2f [ 5692.380486] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5692.386019] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5692.399863] ? watchdog+0x40/0x40 [ 5692.406234] ? hrtimer_interrupt+0x128/0x2c0 [ 5692.411842] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5692.424401] ? apic_timer_interrupt+0xf/0x20 [ 5692.426171] [ 5692.435892] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5692.454228] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5692.464286] do_raw_spin_lock+0x177/0x190 [ 5692.471396] _raw_spin_lock+0x21/0x30 [ 5692.478085] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5692.486356] process_one_work+0x2c8/0x700 [ 5692.493824] worker_thread+0x296/0x6e0 [ 5692.496066] ? rescuer_thread+0x570/0x570 [ 5692.503340] kthread+0x1d1/0x200 [ 5692.508911] ? set_kthread_struct+0x70/0x70 [ 5692.516519] ret_from_fork+0x1f/0x30 [ 5708.052371] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 5708.057378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5708.127236] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5708.150954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5708.169458] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5708.174491] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5708.202733] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5708.219148] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5708.234544] RDX: 00000000000027e1 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5708.246714] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5708.265204] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5708.272336] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5708.280313] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5708.302083] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5708.312275] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5708.322219] Call Trace: [ 5708.332091] [ 5708.334734] ? show_regs.cold.9+0x22/0x2f [ 5708.341965] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5708.352944] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5708.356638] ? watchdog+0x40/0x40 [ 5708.357796] ? hrtimer_interrupt+0x128/0x2c0 [ 5708.366528] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5708.373560] ? apic_timer_interrupt+0xf/0x20 [ 5708.379651] [ 5708.383708] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5708.387413] do_raw_spin_lock+0x177/0x190 [ 5708.393136] _raw_spin_lock+0x21/0x30 [ 5708.398088] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5708.413512] ? do_raw_spin_unlock+0x75/0x190 [ 5708.417365] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5708.423665] ? do_wait_intr+0xf0/0xf0 [ 5708.435332] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5708.442678] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5708.449117] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5708.467285] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5708.481754] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5708.493460] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5708.501078] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5708.516599] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5708.526054] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5708.532059] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5708.540699] lmv_intent_open+0x338/0x1150 [lmv] [ 5708.542325] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5708.552128] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5708.563299] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5708.567317] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5708.576295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5708.597286] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5708.612486] ? do_raw_spin_unlock+0x75/0x190 [ 5708.617664] ? _raw_spin_unlock+0x12/0x30 [ 5708.626763] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5708.637406] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5708.647942] ll_atomic_open+0x36a/0xd70 [lustre] [ 5708.652876] lookup_open+0x356/0xb40 [ 5708.654095] do_last+0x313/0xfc0 [ 5708.672680] ? nd_jump_root+0xe5/0x160 [ 5708.685943] ? path_init+0x437/0x520 [ 5708.702048] path_openat+0xf7/0x500 [ 5708.709603] ? do_raw_spin_unlock+0x75/0x190 [ 5708.721944] do_filp_open+0x99/0x140 [ 5708.725964] ? getname_flags+0x6e/0x330 [ 5708.730073] ? __check_object_size+0xff/0x256 [ 5708.738017] ? do_raw_spin_unlock+0x75/0x190 [ 5708.743257] ? _raw_spin_unlock+0x12/0x30 [ 5708.758031] do_sys_openat2+0x2b4/0x410 [ 5708.761300] do_sys_open+0x73/0xa0 [ 5708.776148] __x64_sys_openat+0x24/0x30 [ 5708.785539] do_syscall_64+0xc1/0x440 [ 5708.798470] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5708.812432] RIP: 0033:0x7fdc0afbf332 [ 5708.822097] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5708.854666] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5708.870231] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5708.881085] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5708.886604] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5708.892345] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5708.902031] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5716.051762] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5716.054845] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5716.062107] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5716.062189] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5716.062196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5716.062200] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5716.062214] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5716.062219] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5716.062224] RAX: 000000000000748d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5716.062226] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5716.062228] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5716.062231] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5716.062233] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5716.062237] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5716.062239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5716.062242] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5716.062251] Call Trace: [ 5716.062256] [ 5716.062262] ? show_regs.cold.9+0x22/0x2f [ 5716.062274] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5716.062280] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5716.062285] ? watchdog+0x40/0x40 [ 5716.062289] ? hrtimer_interrupt+0x128/0x2c0 [ 5716.062293] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5716.062301] ? apic_timer_interrupt+0xf/0x20 [ 5716.062305] [ 5716.062308] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5716.062312] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5716.062316] ? dump_stack+0xf6/0x10e [ 5716.062322] do_raw_spin_lock+0x177/0x190 [ 5716.062326] _raw_spin_lock+0x21/0x30 [ 5716.062332] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5716.062621] obd_mod_free+0x46/0x140 [ptlrpc] [ 5716.062830] mdc_commit_open+0x91/0xb0 [mdc] [ 5716.063063] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5716.063270] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5716.063464] after_reply+0xe2c/0x1780 [ptlrpc] [ 5716.063661] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5716.065220] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5716.065485] ? do_wait_intr+0xf0/0xf0 [ 5716.065497] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5716.065715] kthread+0x1d1/0x200 [ 5716.065723] ? set_kthread_struct+0x70/0x70 [ 5716.065727] ret_from_fork+0x1f/0x30 [ 5716.436868] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5716.490151] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5716.499424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5716.503831] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5716.508208] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5716.519570] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5716.529098] RAX: 0000000000000512 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5716.546122] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5716.551862] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5716.555197] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5716.558924] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5716.565561] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5716.571736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5716.573646] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5716.581916] Call Trace: [ 5716.587655] [ 5716.588875] ? show_regs.cold.9+0x22/0x2f [ 5716.594032] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5716.598863] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5716.602840] ? watchdog+0x40/0x40 [ 5716.604812] ? hrtimer_interrupt+0x128/0x2c0 [ 5716.608073] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5716.610583] ? apic_timer_interrupt+0xf/0x20 [ 5716.615845] [ 5716.618229] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5716.621199] do_raw_spin_lock+0x177/0x190 [ 5716.622679] _raw_spin_lock+0x21/0x30 [ 5716.627056] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5716.632195] ? do_raw_spin_unlock+0x75/0x190 [ 5716.633679] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5716.637513] ? do_wait_intr+0xf0/0xf0 [ 5716.639460] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5716.644653] mdc_close+0x53b/0xd70 [mdc] [ 5716.651294] lmv_close+0x18d/0x490 [lmv] [ 5716.657079] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5716.662563] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5716.668147] ll_md_close+0x274/0x800 [lustre] [ 5716.673998] ll_file_release+0x128/0xc50 [lustre] [ 5716.679780] __fput+0x10d/0x440 [ 5716.680740] ____fput+0x12/0x20 [ 5716.681754] task_work_run+0xc6/0x110 [ 5716.687349] exit_to_usermode_loop+0x1dd/0x1f0 [ 5716.693826] do_syscall_64+0x3d6/0x440 [ 5716.698123] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5716.705898] RIP: 0033:0x7fdc0afbfcd8 [ 5716.710399] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5716.725630] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5716.735914] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5716.743900] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5716.747438] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5716.757410] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5716.762170] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5720.053144] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5720.061273] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5720.108057] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5720.114985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5720.122495] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5720.130397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5720.138023] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5720.151543] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5720.155711] RAX: 0000000000007f90 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5720.159098] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5720.161007] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5720.165012] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5720.169531] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5720.173196] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5720.178507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5720.193717] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5720.201019] Call Trace: [ 5720.202163] [ 5720.203986] ? show_regs.cold.9+0x22/0x2f [ 5720.206764] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5720.222746] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5720.232779] ? watchdog+0x40/0x40 [ 5720.239364] ? hrtimer_interrupt+0x128/0x2c0 [ 5720.249997] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5720.260940] ? apic_timer_interrupt+0xf/0x20 [ 5720.269114] [ 5720.270220] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5720.282718] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5720.287981] do_raw_spin_lock+0x177/0x190 [ 5720.289823] _raw_spin_lock+0x21/0x30 [ 5720.296524] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5720.300883] process_one_work+0x2c8/0x700 [ 5720.306190] worker_thread+0x296/0x6e0 [ 5720.316148] ? rescuer_thread+0x570/0x570 [ 5720.318975] kthread+0x1d1/0x200 [ 5720.323909] ? set_kthread_struct+0x70/0x70 [ 5720.329779] ret_from_fork+0x1f/0x30 [ 5720.543034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3695s! [ 5720.548052] Showing busy workqueues and worker pools: [ 5720.555603] workqueue events: flags=0x0 [ 5720.561074] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5720.570216] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5720.575912] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5720.582944] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5720.599051] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5720.611982] pending: kfree_rcu_monitor [ 5720.617496] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5720.625320] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5720.643097] workqueue events_long: flags=0x0 [ 5720.646024] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5720.659623] pending: lu_objects_flush [obdclass] [ 5720.666353] workqueue events_unbound: flags=0x2 [ 5720.669563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5720.680817] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5720.697836] workqueue events_power_efficient: flags=0x80 [ 5720.703299] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5720.711751] pending: neigh_periodic_work, check_lifetime [ 5720.721024] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5720.736199] pending: do_cache_clean, neigh_periodic_work [ 5720.742897] workqueue mm_percpu_wq: flags=0x8 [ 5720.748569] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5720.755599] pending: lru_add_drain_per_cpu [ 5720.769685] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5720.782547] pending: vmstat_update, lru_add_drain_per_cpu [ 5720.788123] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5720.793418] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5720.800650] workqueue writeback: flags=0x4a [ 5720.806997] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5720.814607] pending: wb_workfn, wb_workfn [ 5720.819312] workqueue xprtiod: flags=0xa [ 5720.824538] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5720.832611] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5720.841867] workqueue ipv6_addrconf: flags=0x40008 [ 5720.849288] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5720.859028] pending: addrconf_verify_work [ 5720.864152] workqueue knbd0-recv: flags=0x1a [ 5720.869017] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5720.873859] in-flight: 655:recv_work [ 5720.879962] workqueue ptlrpc_pinger: flags=0x2 [ 5720.882950] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5720.888762] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5720.897840] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3695s workers=4 idle: 36396 38 11188 [ 5720.906854] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5711s workers=2 idle: 658 [ 5736.052105] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 5736.055656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5736.078318] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5736.085775] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5736.091770] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5736.094461] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5736.099486] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5736.103606] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5736.106491] RDX: 0000000000002608 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5736.116510] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5736.120123] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5736.124936] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5736.130181] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5736.132912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5736.134821] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5736.137545] Call Trace: [ 5736.141544] [ 5736.142620] ? show_regs.cold.9+0x22/0x2f [ 5736.146331] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5736.151953] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5736.155780] ? watchdog+0x40/0x40 [ 5736.158790] ? hrtimer_interrupt+0x128/0x2c0 [ 5736.162352] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5736.165628] ? apic_timer_interrupt+0xf/0x20 [ 5736.167984] [ 5736.168723] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5736.172071] do_raw_spin_lock+0x177/0x190 [ 5736.173929] _raw_spin_lock+0x21/0x30 [ 5736.175824] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5736.179450] ? do_raw_spin_unlock+0x75/0x190 [ 5736.181753] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5736.185803] ? do_wait_intr+0xf0/0xf0 [ 5736.188859] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5736.190752] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5736.194386] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5736.197643] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5736.202560] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5736.205392] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5736.207744] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5736.208903] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5736.210931] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5736.216788] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5736.222928] lmv_intent_open+0x338/0x1150 [lmv] [ 5736.227354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5736.229346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5736.231683] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5736.233258] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5736.235390] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5736.237688] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5736.239454] ? do_raw_spin_unlock+0x75/0x190 [ 5736.240735] ? _raw_spin_unlock+0x12/0x30 [ 5736.242746] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5736.245578] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5736.247695] ll_atomic_open+0x36a/0xd70 [lustre] [ 5736.249974] lookup_open+0x356/0xb40 [ 5736.251174] do_last+0x313/0xfc0 [ 5736.252194] ? nd_jump_root+0xe5/0x160 [ 5736.253324] ? path_init+0x437/0x520 [ 5736.254640] path_openat+0xf7/0x500 [ 5736.256129] ? do_raw_spin_unlock+0x75/0x190 [ 5736.257942] do_filp_open+0x99/0x140 [ 5736.259440] ? getname_flags+0x6e/0x330 [ 5736.261810] ? __check_object_size+0xff/0x256 [ 5736.263783] ? do_raw_spin_unlock+0x75/0x190 [ 5736.266337] ? _raw_spin_unlock+0x12/0x30 [ 5736.267333] do_sys_openat2+0x2b4/0x410 [ 5736.268268] do_sys_open+0x73/0xa0 [ 5736.269058] __x64_sys_openat+0x24/0x30 [ 5736.271259] do_syscall_64+0xc1/0x440 [ 5736.273474] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5736.275204] RIP: 0033:0x7fdc0afbf332 [ 5736.276636] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5736.290794] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5736.296791] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5736.303954] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5736.311347] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5736.319243] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5736.323524] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5744.054041] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5744.056711] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5744.097162] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5744.109552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5744.115832] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5744.117858] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5744.126910] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5744.129933] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5744.134143] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5744.138592] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5744.144861] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5744.149217] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5744.152218] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5744.156163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5744.158629] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5744.164153] Call Trace: [ 5744.166068] [ 5744.166648] ? show_regs.cold.9+0x22/0x2f [ 5744.167901] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5744.170527] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5744.174154] ? watchdog+0x40/0x40 [ 5744.175533] ? hrtimer_interrupt+0x128/0x2c0 [ 5744.177826] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5744.186724] ? apic_timer_interrupt+0xf/0x20 [ 5744.191115] [ 5744.192358] ? native_safe_halt+0x2e/0x50 [ 5744.195736] kvm_wait+0xcd/0xe0 [ 5744.199463] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5744.202313] do_raw_spin_lock+0x177/0x190 [ 5744.208343] _raw_spin_lock+0x21/0x30 [ 5744.210026] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5744.216305] ? do_raw_spin_unlock+0x75/0x190 [ 5744.219632] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5744.228391] ? do_wait_intr+0xf0/0xf0 [ 5744.231961] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5744.234690] mdc_close+0x53b/0xd70 [mdc] [ 5744.239241] lmv_close+0x18d/0x490 [lmv] [ 5744.240731] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5744.245933] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5744.247707] ll_md_close+0x274/0x800 [lustre] [ 5744.253087] ll_file_release+0x128/0xc50 [lustre] [ 5744.260142] __fput+0x10d/0x440 [ 5744.263296] ____fput+0x12/0x20 [ 5744.265262] task_work_run+0xc6/0x110 [ 5744.268901] exit_to_usermode_loop+0x1dd/0x1f0 [ 5744.270271] do_syscall_64+0x3d6/0x440 [ 5744.273402] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5744.275315] RIP: 0033:0x7fdc0afbfcd8 [ 5744.277090] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5744.284000] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5744.286152] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5744.288310] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5744.290605] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5744.293234] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5744.295693] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5748.056098] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 5748.056299] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 5748.061936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5748.062023] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5748.062030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5748.062037] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5748.062362] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5748.062374] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5748.062379] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5748.062383] RAX: 0000000000006a01 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5748.062385] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5748.062387] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5748.062389] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5748.062392] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5748.062396] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5748.062398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5748.062401] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5748.062409] Call Trace: [ 5748.062415] [ 5748.062421] ? show_regs.cold.9+0x22/0x2f [ 5748.062432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5748.062438] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5748.062449] ? watchdog+0x40/0x40 [ 5748.062453] ? hrtimer_interrupt+0x128/0x2c0 [ 5748.062458] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5748.062466] ? apic_timer_interrupt+0xf/0x20 [ 5748.062471] [ 5748.062474] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5748.062479] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5748.062484] do_raw_spin_lock+0x177/0x190 [ 5748.062489] _raw_spin_lock+0x21/0x30 [ 5748.062494] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5748.062795] process_one_work+0x2c8/0x700 [ 5748.062801] worker_thread+0x296/0x6e0 [ 5748.062819] ? rescuer_thread+0x570/0x570 [ 5748.062824] kthread+0x1d1/0x200 [ 5748.062828] ? set_kthread_struct+0x70/0x70 [ 5748.062832] ret_from_fork+0x1f/0x30 [ 5748.287240] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5748.317354] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5748.327653] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5748.332996] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5748.335000] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5748.351155] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5748.359933] RAX: 000000000000640d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5748.363879] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5748.368271] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5748.377696] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5748.381308] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5748.386883] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5748.390504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5748.394325] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5748.397326] Call Trace: [ 5748.399048] [ 5748.400401] ? show_regs.cold.9+0x22/0x2f [ 5748.404953] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5748.409081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5748.411354] ? watchdog+0x40/0x40 [ 5748.417362] ? hrtimer_interrupt+0x128/0x2c0 [ 5748.420324] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5748.423299] ? apic_timer_interrupt+0xf/0x20 [ 5748.426009] [ 5748.427038] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5748.430300] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5748.434017] ? dump_stack+0xf6/0x10e [ 5748.435990] do_raw_spin_lock+0x177/0x190 [ 5748.439375] _raw_spin_lock+0x21/0x30 [ 5748.443025] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5748.448164] obd_mod_free+0x46/0x140 [ptlrpc] [ 5748.454901] mdc_commit_open+0x91/0xb0 [mdc] [ 5748.458111] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5748.464407] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5748.474262] after_reply+0xe2c/0x1780 [ptlrpc] [ 5748.487126] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5748.498010] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5748.501103] ? do_wait_intr+0xf0/0xf0 [ 5748.503261] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5748.509266] kthread+0x1d1/0x200 [ 5748.512319] ? set_kthread_struct+0x70/0x70 [ 5748.513835] ret_from_fork+0x1f/0x30 [ 5751.263144] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3726s! [ 5751.268617] Showing busy workqueues and worker pools: [ 5751.271612] workqueue events: flags=0x0 [ 5751.274091] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5751.278169] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5751.282313] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5751.285893] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5751.291091] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5751.294330] pending: kfree_rcu_monitor [ 5751.296233] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5751.298020] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5751.308115] workqueue events_long: flags=0x0 [ 5751.309612] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5751.312011] pending: lu_objects_flush [obdclass] [ 5751.313979] workqueue events_unbound: flags=0x2 [ 5751.315268] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5751.317251] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5751.320180] workqueue events_power_efficient: flags=0x80 [ 5751.322187] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5751.324246] pending: neigh_periodic_work, check_lifetime [ 5751.326241] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5751.327931] pending: do_cache_clean, neigh_periodic_work [ 5751.329457] workqueue mm_percpu_wq: flags=0x8 [ 5751.330634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5751.332494] pending: lru_add_drain_per_cpu [ 5751.333671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5751.337411] pending: vmstat_update, lru_add_drain_per_cpu [ 5751.340303] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5751.344803] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5751.349357] workqueue writeback: flags=0x4a [ 5751.351212] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5751.354100] pending: wb_workfn, wb_workfn [ 5751.355580] workqueue xprtiod: flags=0xa [ 5751.356890] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5751.359676] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5751.363393] workqueue ipv6_addrconf: flags=0x40008 [ 5751.365722] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5751.368837] pending: addrconf_verify_work [ 5751.370359] workqueue knbd0-recv: flags=0x1a [ 5751.371897] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5751.374830] in-flight: 655:recv_work [ 5751.376759] workqueue ptlrpc_pinger: flags=0x2 [ 5751.379283] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5751.382619] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5751.385885] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3726s workers=4 idle: 36396 38 11188 [ 5751.389410] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5741s workers=2 idle: 658 [ 5764.052132] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 5764.055287] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5764.078670] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5764.083902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5764.089620] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5764.091606] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5764.099087] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5764.101877] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5764.104533] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5764.107628] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5764.110129] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5764.112868] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5764.115788] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5764.118762] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5764.120815] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5764.123281] Call Trace: [ 5764.124214] [ 5764.124829] ? show_regs.cold.9+0x22/0x2f [ 5764.126202] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5764.129261] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5764.130847] ? watchdog+0x40/0x40 [ 5764.131871] ? hrtimer_interrupt+0x128/0x2c0 [ 5764.133312] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5764.134741] ? apic_timer_interrupt+0xf/0x20 [ 5764.136066] [ 5764.136648] ? native_safe_halt+0x2e/0x50 [ 5764.138046] kvm_wait+0xcd/0xe0 [ 5764.138882] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5764.140434] do_raw_spin_lock+0x177/0x190 [ 5764.141906] _raw_spin_lock+0x21/0x30 [ 5764.143170] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5764.144946] ? do_raw_spin_unlock+0x75/0x190 [ 5764.146137] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5764.147773] ? do_wait_intr+0xf0/0xf0 [ 5764.149064] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5764.150619] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5764.152168] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5764.153942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5764.155576] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5764.157396] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5764.158794] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5764.160681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5764.164626] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5764.167850] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5764.170412] lmv_intent_open+0x338/0x1150 [lmv] [ 5764.172587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5764.175170] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5764.177903] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5764.180234] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5764.182438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5764.184763] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5764.187067] ? do_raw_spin_unlock+0x75/0x190 [ 5764.188725] ? _raw_spin_unlock+0x12/0x30 [ 5764.190391] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5764.192281] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5764.195411] ll_atomic_open+0x36a/0xd70 [lustre] [ 5764.199652] lookup_open+0x356/0xb40 [ 5764.202592] do_last+0x313/0xfc0 [ 5764.203952] ? nd_jump_root+0xe5/0x160 [ 5764.207183] ? path_init+0x437/0x520 [ 5764.209108] path_openat+0xf7/0x500 [ 5764.210479] ? do_raw_spin_unlock+0x75/0x190 [ 5764.211981] do_filp_open+0x99/0x140 [ 5764.214480] ? getname_flags+0x6e/0x330 [ 5764.216592] ? __check_object_size+0xff/0x256 [ 5764.221094] ? do_raw_spin_unlock+0x75/0x190 [ 5764.224347] ? _raw_spin_unlock+0x12/0x30 [ 5764.228135] do_sys_openat2+0x2b4/0x410 [ 5764.230180] do_sys_open+0x73/0xa0 [ 5764.232159] __x64_sys_openat+0x24/0x30 [ 5764.234210] do_syscall_64+0xc1/0x440 [ 5764.236773] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5764.242713] RIP: 0033:0x7fdc0afbf332 [ 5764.244052] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5764.255528] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5764.259016] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5764.262004] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5764.265403] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5764.269194] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5764.273201] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5772.054020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5772.057262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5772.096351] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5772.116536] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5772.129237] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5772.136029] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5772.161999] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5772.167895] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5772.176050] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5772.185503] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5772.200773] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5772.205131] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5772.220314] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5772.223303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5772.233153] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5772.244132] Call Trace: [ 5772.246278] [ 5772.249508] ? show_regs.cold.9+0x22/0x2f [ 5772.252273] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5772.264122] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5772.274585] ? watchdog+0x40/0x40 [ 5772.275896] ? hrtimer_interrupt+0x128/0x2c0 [ 5772.281431] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5772.288176] ? apic_timer_interrupt+0xf/0x20 [ 5772.296857] [ 5772.298888] ? native_safe_halt+0x2e/0x50 [ 5772.309636] kvm_wait+0xcd/0xe0 [ 5772.322925] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5772.331945] do_raw_spin_lock+0x177/0x190 [ 5772.338149] _raw_spin_lock+0x21/0x30 [ 5772.341225] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5772.345492] ? do_raw_spin_unlock+0x75/0x190 [ 5772.348028] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5772.355662] ? do_wait_intr+0xf0/0xf0 [ 5772.357586] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5772.361173] mdc_close+0x53b/0xd70 [mdc] [ 5772.363721] lmv_close+0x18d/0x490 [lmv] [ 5772.367805] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5772.371395] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5772.375699] ll_md_close+0x274/0x800 [lustre] [ 5772.383599] ll_file_release+0x128/0xc50 [lustre] [ 5772.389617] __fput+0x10d/0x440 [ 5772.394588] ____fput+0x12/0x20 [ 5772.398162] task_work_run+0xc6/0x110 [ 5772.399267] exit_to_usermode_loop+0x1dd/0x1f0 [ 5772.410204] do_syscall_64+0x3d6/0x440 [ 5772.411891] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5772.421935] RIP: 0033:0x7fdc0afbfcd8 [ 5772.425905] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5772.455822] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5772.466609] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5772.473904] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5772.476343] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5772.483822] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5772.500278] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5776.048029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5776.051122] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5776.052213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 5776.058240] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [ 5776.083248] intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5776.083285] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5776.083292] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5776.083295] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5776.083366] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5776.083372] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5776.083378] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5776.083677] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5776.083680] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5776.083682] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5776.083685] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5776.083688] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5776.083691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5776.083693] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5776.083702] Call Trace: [ 5776.083708] [ 5776.083714] ? show_regs.cold.9+0x22/0x2f [ 5776.083816] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5776.083823] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5776.083827] ? watchdog+0x40/0x40 [ 5776.083832] ? hrtimer_interrupt+0x128/0x2c0 [ 5776.083837] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5776.083844] ? apic_timer_interrupt+0xf/0x20 [ 5776.083848] [ 5776.083852] ? native_safe_halt+0x2e/0x50 [ 5776.083856] kvm_wait+0xcd/0xe0 [ 5776.083864] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5776.083871] ? dump_stack+0xf6/0x10e [ 5776.083877] do_raw_spin_lock+0x177/0x190 [ 5776.083882] _raw_spin_lock+0x21/0x30 [ 5776.083889] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5776.104847] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5776.104891] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5776.105024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5776.105034] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5776.105301] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5776.105313] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5776.105317] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5776.105321] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5776.105323] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5776.105325] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5776.105327] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5776.105340] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5776.105345] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5776.105348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5776.105350] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5776.105358] Call Trace: [ 5776.105364] [ 5776.105369] ? show_regs.cold.9+0x22/0x2f [ 5776.105379] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5776.105386] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5776.105390] ? watchdog+0x40/0x40 [ 5776.105395] ? hrtimer_interrupt+0x128/0x2c0 [ 5776.105399] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5776.105405] ? apic_timer_interrupt+0xf/0x20 [ 5776.105410] [ 5776.105413] ? native_safe_halt+0x2e/0x50 [ 5776.105417] kvm_wait+0xcd/0xe0 [ 5776.105424] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5776.105743] do_raw_spin_lock+0x177/0x190 [ 5776.105751] _raw_spin_lock+0x21/0x30 [ 5776.105755] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5776.106116] process_one_work+0x2c8/0x700 [ 5776.106123] worker_thread+0x296/0x6e0 [ 5776.106128] ? rescuer_thread+0x570/0x570 [ 5776.106132] kthread+0x1d1/0x200 [ 5776.106136] ? set_kthread_struct+0x70/0x70 [ 5776.106140] ret_from_fork+0x1f/0x30 [ 5776.402080] obd_mod_free+0x46/0x140 [ptlrpc] [ 5776.404243] mdc_commit_open+0x91/0xb0 [mdc] [ 5776.406162] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5776.408283] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5776.410865] after_reply+0xe2c/0x1780 [ptlrpc] [ 5776.413103] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5776.416282] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5776.419838] ? do_wait_intr+0xf0/0xf0 [ 5776.422220] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5776.426671] kthread+0x1d1/0x200 [ 5776.429576] ? set_kthread_struct+0x70/0x70 [ 5776.434118] ret_from_fork+0x1f/0x30 [ 5781.471513] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3756s! [ 5781.484566] Showing busy workqueues and worker pools: [ 5781.488371] workqueue events: flags=0x0 [ 5781.490527] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5781.497165] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5781.500929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5781.512783] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5781.520414] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5781.525237] pending: kfree_rcu_monitor [ 5781.527008] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5781.541713] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5781.549570] workqueue events_long: flags=0x0 [ 5781.555412] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5781.559196] pending: lu_objects_flush [obdclass] [ 5781.562052] workqueue events_unbound: flags=0x2 [ 5781.563926] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5781.573752] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5781.584845] workqueue events_power_efficient: flags=0x80 [ 5781.588839] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5781.593480] pending: neigh_periodic_work, check_lifetime [ 5781.599332] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5781.602470] pending: do_cache_clean, neigh_periodic_work [ 5781.605172] workqueue mm_percpu_wq: flags=0x8 [ 5781.607791] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5781.621986] pending: lru_add_drain_per_cpu [ 5781.627693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5781.634597] pending: vmstat_update, lru_add_drain_per_cpu [ 5781.636580] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5781.642087] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5781.645615] workqueue writeback: flags=0x4a [ 5781.647755] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5781.653961] pending: wb_workfn, wb_workfn [ 5781.656922] workqueue xprtiod: flags=0xa [ 5781.658456] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5781.662748] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5781.670810] workqueue ipv6_addrconf: flags=0x40008 [ 5781.672921] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5781.681931] pending: addrconf_verify_work [ 5781.685852] workqueue knbd0-recv: flags=0x1a [ 5781.690688] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5781.693755] in-flight: 655:recv_work [ 5781.695875] workqueue ptlrpc_pinger: flags=0x2 [ 5781.697296] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5781.704945] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5781.709170] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3756s workers=4 idle: 36396 38 11188 [ 5781.717035] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5772s workers=2 idle: 658 [ 5792.052022] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 5792.062274] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5792.098282] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5792.112555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5792.118860] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5792.120190] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5792.144845] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5792.154400] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5792.161033] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5792.171122] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5792.179026] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5792.184017] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5792.189917] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5792.201770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5792.207837] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5792.215742] Call Trace: [ 5792.222724] [ 5792.224008] ? show_regs.cold.9+0x22/0x2f [ 5792.226954] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5792.230924] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5792.234900] ? watchdog+0x40/0x40 [ 5792.238932] ? hrtimer_interrupt+0x128/0x2c0 [ 5792.242885] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5792.249204] ? apic_timer_interrupt+0xf/0x20 [ 5792.253944] [ 5792.255926] ? native_safe_halt+0x2e/0x50 [ 5792.259189] kvm_wait+0xcd/0xe0 [ 5792.261380] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5792.266958] do_raw_spin_lock+0x177/0x190 [ 5792.273039] _raw_spin_lock+0x21/0x30 [ 5792.278075] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5792.286774] ? do_raw_spin_unlock+0x75/0x190 [ 5792.292767] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5792.298100] ? do_wait_intr+0xf0/0xf0 [ 5792.302079] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5792.307379] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5792.313140] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5792.316325] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5792.320151] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5792.324747] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5792.327980] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5792.331465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5792.336251] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5792.339804] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5792.343833] lmv_intent_open+0x338/0x1150 [lmv] [ 5792.346565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5792.350577] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5792.354771] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5792.360712] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5792.362877] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5792.367663] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5792.372196] ? do_raw_spin_unlock+0x75/0x190 [ 5792.381758] ? _raw_spin_unlock+0x12/0x30 [ 5792.384129] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5792.391687] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5792.397405] ll_atomic_open+0x36a/0xd70 [lustre] [ 5792.401659] lookup_open+0x356/0xb40 [ 5792.403646] do_last+0x313/0xfc0 [ 5792.404926] ? nd_jump_root+0xe5/0x160 [ 5792.410110] ? path_init+0x437/0x520 [ 5792.416628] path_openat+0xf7/0x500 [ 5792.421364] ? do_raw_spin_unlock+0x75/0x190 [ 5792.425123] do_filp_open+0x99/0x140 [ 5792.427672] ? getname_flags+0x6e/0x330 [ 5792.429888] ? __check_object_size+0xff/0x256 [ 5792.437100] ? do_raw_spin_unlock+0x75/0x190 [ 5792.440189] ? _raw_spin_unlock+0x12/0x30 [ 5792.443001] do_sys_openat2+0x2b4/0x410 [ 5792.444408] do_sys_open+0x73/0xa0 [ 5792.452535] __x64_sys_openat+0x24/0x30 [ 5792.454495] do_syscall_64+0xc1/0x440 [ 5792.463139] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5792.464882] RIP: 0033:0x7fdc0afbf332 [ 5792.466188] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5792.493706] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5792.502410] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5792.509601] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5792.517126] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5792.524884] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5792.529894] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5800.054022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5800.058915] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5800.102402] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5800.111180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5800.119210] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5800.124728] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5800.146195] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5800.152199] RAX: 0000000000001315 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5800.156116] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5800.160116] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5800.163422] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5800.166451] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5800.169439] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5800.172875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5800.175539] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5800.179114] Call Trace: [ 5800.180170] [ 5800.181031] ? show_regs.cold.9+0x22/0x2f [ 5800.182787] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5800.184882] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5800.186814] ? watchdog+0x40/0x40 [ 5800.188554] ? hrtimer_interrupt+0x128/0x2c0 [ 5800.190448] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5800.196760] ? apic_timer_interrupt+0xf/0x20 [ 5800.199813] [ 5800.200657] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5800.204158] do_raw_spin_lock+0x177/0x190 [ 5800.207817] _raw_spin_lock+0x21/0x30 [ 5800.211466] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5800.218124] ? do_raw_spin_unlock+0x75/0x190 [ 5800.222861] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5800.226633] ? do_wait_intr+0xf0/0xf0 [ 5800.230396] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5800.234514] mdc_close+0x53b/0xd70 [mdc] [ 5800.237520] lmv_close+0x18d/0x490 [lmv] [ 5800.242488] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5800.251019] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5800.254507] ll_md_close+0x274/0x800 [lustre] [ 5800.256473] ll_file_release+0x128/0xc50 [lustre] [ 5800.258514] __fput+0x10d/0x440 [ 5800.259873] ____fput+0x12/0x20 [ 5800.261088] task_work_run+0xc6/0x110 [ 5800.262251] exit_to_usermode_loop+0x1dd/0x1f0 [ 5800.264055] do_syscall_64+0x3d6/0x440 [ 5800.265517] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5800.267432] RIP: 0033:0x7fdc0afbfcd8 [ 5800.268673] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5800.279953] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5800.289486] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5800.295499] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5800.303578] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5800.311389] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5800.322871] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5804.051163] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5804.055152] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5804.101210] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5804.111590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5804.122660] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5804.126566] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5804.130410] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5804.142311] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5804.146750] RAX: 0000000000003984 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5804.151090] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5804.155945] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5804.160675] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5804.165250] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5804.169224] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5804.174550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5804.178197] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5804.181748] Call Trace: [ 5804.183341] [ 5804.184279] ? show_regs.cold.9+0x22/0x2f [ 5804.186789] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5804.189249] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5804.191804] ? watchdog+0x40/0x40 [ 5804.193563] ? hrtimer_interrupt+0x128/0x2c0 [ 5804.196389] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5804.198527] ? apic_timer_interrupt+0xf/0x20 [ 5804.203259] [ 5804.204314] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5804.208050] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5804.210927] do_raw_spin_lock+0x177/0x190 [ 5804.213378] _raw_spin_lock+0x21/0x30 [ 5804.215352] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5804.218425] process_one_work+0x2c8/0x700 [ 5804.220398] worker_thread+0x296/0x6e0 [ 5804.223337] ? rescuer_thread+0x570/0x570 [ 5804.226367] kthread+0x1d1/0x200 [ 5804.228060] ? set_kthread_struct+0x70/0x70 [ 5804.230641] ret_from_fork+0x1f/0x30 [ 5808.048076] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 5808.050659] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5808.099317] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5808.111130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5808.120069] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5808.126406] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5808.145130] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5808.152654] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5808.159482] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5808.168244] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5808.176116] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5808.182216] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5808.190176] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5808.198647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5808.203503] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5808.212905] Call Trace: [ 5808.214114] [ 5808.216478] ? show_regs.cold.9+0x22/0x2f [ 5808.220089] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5808.226719] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5808.231184] ? watchdog+0x40/0x40 [ 5808.234383] ? hrtimer_interrupt+0x128/0x2c0 [ 5808.235949] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5808.242733] ? apic_timer_interrupt+0xf/0x20 [ 5808.247693] [ 5808.249926] ? native_safe_halt+0x2e/0x50 [ 5808.252693] kvm_wait+0xcd/0xe0 [ 5808.258188] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5808.267919] ? dump_stack+0xf6/0x10e [ 5808.271997] do_raw_spin_lock+0x177/0x190 [ 5808.276299] _raw_spin_lock+0x21/0x30 [ 5808.279944] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5808.287189] obd_mod_free+0x46/0x140 [ptlrpc] [ 5808.293468] mdc_commit_open+0x91/0xb0 [mdc] [ 5808.297509] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5808.305448] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5808.313706] after_reply+0xe2c/0x1780 [ptlrpc] [ 5808.317823] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5808.322268] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5808.327877] ? do_wait_intr+0xf0/0xf0 [ 5808.331258] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5808.336383] kthread+0x1d1/0x200 [ 5808.340725] ? set_kthread_struct+0x70/0x70 [ 5808.342292] ret_from_fork+0x1f/0x30 [ 5812.191029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3787s! [ 5812.200593] Showing busy workqueues and worker pools: [ 5812.204446] workqueue events: flags=0x0 [ 5812.207545] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5812.211285] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5812.218049] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5812.221779] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5812.228245] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5812.233284] pending: kfree_rcu_monitor [ 5812.238650] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5812.245713] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5812.256008] workqueue events_long: flags=0x0 [ 5812.258420] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5812.261308] pending: lu_objects_flush [obdclass] [ 5812.265015] workqueue events_unbound: flags=0x2 [ 5812.267123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5812.270372] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5812.277099] workqueue events_power_efficient: flags=0x80 [ 5812.282149] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5812.287946] pending: neigh_periodic_work, check_lifetime [ 5812.290925] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5812.295363] pending: do_cache_clean, neigh_periodic_work [ 5812.302196] workqueue mm_percpu_wq: flags=0x8 [ 5812.305205] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5812.309151] pending: lru_add_drain_per_cpu [ 5812.311245] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5812.314896] pending: vmstat_update, lru_add_drain_per_cpu [ 5812.317645] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5812.320767] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5812.324296] workqueue writeback: flags=0x4a [ 5812.326391] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5812.330155] pending: wb_workfn, wb_workfn [ 5812.332434] workqueue xprtiod: flags=0xa [ 5812.334499] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5812.337501] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5812.341887] workqueue ipv6_addrconf: flags=0x40008 [ 5812.344222] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5812.347374] pending: addrconf_verify_work [ 5812.350105] workqueue knbd0-recv: flags=0x1a [ 5812.352592] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5812.355636] in-flight: 655:recv_work [ 5812.357671] workqueue ptlrpc_pinger: flags=0x2 [ 5812.360066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5812.362620] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5812.366568] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3787s workers=4 idle: 36396 38 11188 [ 5812.370430] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5802s workers=2 idle: 658 [ 5820.053236] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 5820.063342] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5820.108948] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5820.113828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5820.117559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5820.126349] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5820.134170] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5820.136393] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5820.139028] RDX: 0000000000000db3 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5820.140904] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5820.143260] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5820.145286] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5820.147441] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5820.149873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5820.151931] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5820.154920] Call Trace: [ 5820.155629] [ 5820.156222] ? show_regs.cold.9+0x22/0x2f [ 5820.157314] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5820.158686] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5820.160086] ? watchdog+0x40/0x40 [ 5820.161217] ? hrtimer_interrupt+0x128/0x2c0 [ 5820.162685] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5820.164199] ? apic_timer_interrupt+0xf/0x20 [ 5820.165463] [ 5820.166212] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5820.167737] do_raw_spin_lock+0x177/0x190 [ 5820.169162] _raw_spin_lock+0x21/0x30 [ 5820.174184] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5820.177273] ? do_raw_spin_unlock+0x75/0x190 [ 5820.180859] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5820.184014] ? do_wait_intr+0xf0/0xf0 [ 5820.185442] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5820.187649] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5820.194069] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5820.198239] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5820.200137] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5820.204422] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5820.206862] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5820.209851] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5820.212273] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5820.218479] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5820.221203] lmv_intent_open+0x338/0x1150 [lmv] [ 5820.224314] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5820.225804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5820.228991] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5820.237570] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5820.238843] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5820.241786] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5820.244312] ? do_raw_spin_unlock+0x75/0x190 [ 5820.246091] ? _raw_spin_unlock+0x12/0x30 [ 5820.248123] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5820.252983] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5820.262365] ll_atomic_open+0x36a/0xd70 [lustre] [ 5820.263833] lookup_open+0x356/0xb40 [ 5820.264684] do_last+0x313/0xfc0 [ 5820.266924] ? nd_jump_root+0xe5/0x160 [ 5820.267973] ? path_init+0x437/0x520 [ 5820.269157] path_openat+0xf7/0x500 [ 5820.270322] ? do_raw_spin_unlock+0x75/0x190 [ 5820.272007] do_filp_open+0x99/0x140 [ 5820.273294] ? getname_flags+0x6e/0x330 [ 5820.274683] ? __check_object_size+0xff/0x256 [ 5820.276099] ? do_raw_spin_unlock+0x75/0x190 [ 5820.279802] ? _raw_spin_unlock+0x12/0x30 [ 5820.281466] do_sys_openat2+0x2b4/0x410 [ 5820.282977] do_sys_open+0x73/0xa0 [ 5820.284383] __x64_sys_openat+0x24/0x30 [ 5820.285859] do_syscall_64+0xc1/0x440 [ 5820.287355] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5820.294921] RIP: 0033:0x7fdc0afbf332 [ 5820.298916] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5820.310426] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5820.314383] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5820.317716] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5820.324657] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5820.327070] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5820.333924] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5828.054080] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5828.057627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5828.089215] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5828.095270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5828.104244] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5828.107959] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5828.129384] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5828.133899] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5828.136590] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5828.140257] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5828.151712] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5828.155578] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5828.159393] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5828.162938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5828.164755] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5828.176069] Call Trace: [ 5828.180535] [ 5828.181270] ? show_regs.cold.9+0x22/0x2f [ 5828.182634] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5828.187404] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5828.190352] ? watchdog+0x40/0x40 [ 5828.192917] ? hrtimer_interrupt+0x128/0x2c0 [ 5828.199401] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5828.203671] ? apic_timer_interrupt+0xf/0x20 [ 5828.206053] [ 5828.207083] ? native_safe_halt+0x2e/0x50 [ 5828.208721] kvm_wait+0xcd/0xe0 [ 5828.213795] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5828.220186] do_raw_spin_lock+0x177/0x190 [ 5828.223519] _raw_spin_lock+0x21/0x30 [ 5828.227129] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5828.231110] ? do_raw_spin_unlock+0x75/0x190 [ 5828.233754] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5828.239252] ? do_wait_intr+0xf0/0xf0 [ 5828.242928] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5828.247050] mdc_close+0x53b/0xd70 [mdc] [ 5828.249758] lmv_close+0x18d/0x490 [lmv] [ 5828.251957] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5828.256482] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5828.259222] ll_md_close+0x274/0x800 [lustre] [ 5828.261651] ll_file_release+0x128/0xc50 [lustre] [ 5828.264183] __fput+0x10d/0x440 [ 5828.265953] ____fput+0x12/0x20 [ 5828.267304] task_work_run+0xc6/0x110 [ 5828.269220] exit_to_usermode_loop+0x1dd/0x1f0 [ 5828.283761] do_syscall_64+0x3d6/0x440 [ 5828.290920] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5828.293290] RIP: 0033:0x7fdc0afbfcd8 [ 5828.295693] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5828.311490] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5828.317458] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5828.321756] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5828.326704] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5828.332751] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5828.339031] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5832.051483] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5832.058341] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5832.091454] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5832.101765] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5832.108456] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5832.115415] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5832.116948] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5832.129551] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5832.132905] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5832.138767] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5832.141982] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5832.145433] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5832.148810] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5832.151851] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5832.157911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5832.160855] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5832.164520] Call Trace: [ 5832.165801] [ 5832.166971] ? show_regs.cold.9+0x22/0x2f [ 5832.168924] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5832.173996] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5832.177174] ? watchdog+0x40/0x40 [ 5832.178905] ? hrtimer_interrupt+0x128/0x2c0 [ 5832.181227] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5832.183935] ? apic_timer_interrupt+0xf/0x20 [ 5832.186189] [ 5832.187207] ? native_safe_halt+0x2e/0x50 [ 5832.194531] kvm_wait+0xcd/0xe0 [ 5832.196690] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5832.202763] do_raw_spin_lock+0x177/0x190 [ 5832.206656] _raw_spin_lock+0x21/0x30 [ 5832.209353] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5832.212695] process_one_work+0x2c8/0x700 [ 5832.219578] worker_thread+0x296/0x6e0 [ 5832.224317] ? rescuer_thread+0x570/0x570 [ 5832.229411] kthread+0x1d1/0x200 [ 5832.233499] ? set_kthread_struct+0x70/0x70 [ 5832.238232] ret_from_fork+0x1f/0x30 [ 5840.048554] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5840.059577] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5840.096077] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5840.105151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5840.109633] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5840.111098] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5840.120164] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5840.126420] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5840.129086] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5840.135631] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5840.138757] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5840.141443] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5840.147113] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5840.150063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5840.157728] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5840.160780] Call Trace: [ 5840.162731] [ 5840.163447] ? show_regs.cold.9+0x22/0x2f [ 5840.165448] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5840.167513] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5840.168825] ? watchdog+0x40/0x40 [ 5840.170884] ? hrtimer_interrupt+0x128/0x2c0 [ 5840.172463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5840.174644] ? apic_timer_interrupt+0xf/0x20 [ 5840.176481] [ 5840.177343] ? native_safe_halt+0x2e/0x50 [ 5840.178855] kvm_wait+0xcd/0xe0 [ 5840.180133] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5840.182074] ? dump_stack+0xf6/0x10e [ 5840.183569] do_raw_spin_lock+0x177/0x190 [ 5840.184859] _raw_spin_lock+0x21/0x30 [ 5840.185951] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5840.187686] obd_mod_free+0x46/0x140 [ptlrpc] [ 5840.189531] mdc_commit_open+0x91/0xb0 [mdc] [ 5840.190995] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5840.193271] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5840.195403] after_reply+0xe2c/0x1780 [ptlrpc] [ 5840.197466] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5840.199594] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5840.201356] ? do_wait_intr+0xf0/0xf0 [ 5840.202802] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5840.204909] kthread+0x1d1/0x200 [ 5840.206184] ? set_kthread_struct+0x70/0x70 [ 5840.209044] ret_from_fork+0x1f/0x30 [ 5842.399045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3817s! [ 5842.406986] Showing busy workqueues and worker pools: [ 5842.415895] workqueue events: flags=0x0 [ 5842.421544] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5842.428279] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5842.433549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5842.437360] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5842.442888] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5842.445614] pending: kfree_rcu_monitor [ 5842.447989] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5842.453351] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5842.469298] workqueue events_long: flags=0x0 [ 5842.472837] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5842.482824] pending: lu_objects_flush [obdclass] [ 5842.485837] workqueue events_unbound: flags=0x2 [ 5842.489709] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5842.497862] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5842.511296] workqueue events_power_efficient: flags=0x80 [ 5842.515432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5842.525143] pending: neigh_periodic_work, check_lifetime [ 5842.532497] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5842.541418] pending: do_cache_clean, neigh_periodic_work [ 5842.548441] workqueue mm_percpu_wq: flags=0x8 [ 5842.554622] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5842.560403] pending: lru_add_drain_per_cpu [ 5842.564033] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5842.569488] pending: vmstat_update, lru_add_drain_per_cpu [ 5842.577926] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5842.586504] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5842.591384] workqueue writeback: flags=0x4a [ 5842.594316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5842.598143] pending: wb_workfn, wb_workfn [ 5842.604125] workqueue xprtiod: flags=0xa [ 5842.609271] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5842.616016] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5842.621302] workqueue ipv6_addrconf: flags=0x40008 [ 5842.633835] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5842.637955] pending: addrconf_verify_work [ 5842.640135] workqueue knbd0-recv: flags=0x1a [ 5842.642083] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5842.644853] in-flight: 655:recv_work [ 5842.646427] workqueue ptlrpc_pinger: flags=0x2 [ 5842.648474] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5842.651282] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5842.653993] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3817s workers=4 idle: 36396 38 11188 [ 5842.658036] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5833s workers=2 idle: 658 [ 5848.052015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5848.060781] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5848.090041] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5848.103024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5848.113131] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5848.117398] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5848.124771] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5848.128048] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5848.130864] RDX: 000000000000203d RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5848.133879] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5848.136227] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5848.139163] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5848.141927] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5848.148784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5848.150641] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5848.153762] Call Trace: [ 5848.155634] [ 5848.156434] ? show_regs.cold.9+0x22/0x2f [ 5848.158196] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5848.160069] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5848.162827] ? watchdog+0x40/0x40 [ 5848.164336] ? hrtimer_interrupt+0x128/0x2c0 [ 5848.168402] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5848.171788] ? apic_timer_interrupt+0xf/0x20 [ 5848.177976] [ 5848.183392] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5848.185415] do_raw_spin_lock+0x177/0x190 [ 5848.191050] _raw_spin_lock+0x21/0x30 [ 5848.193325] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5848.196351] ? do_raw_spin_unlock+0x75/0x190 [ 5848.199474] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5848.202882] ? do_wait_intr+0xf0/0xf0 [ 5848.204194] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5848.207485] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5848.211635] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5848.215130] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5848.217144] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5848.222518] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5848.227450] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5848.230034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5848.232994] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5848.238820] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5848.243846] lmv_intent_open+0x338/0x1150 [lmv] [ 5848.250287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5848.255822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5848.262834] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5848.266693] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5848.272342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5848.275198] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5848.277994] ? do_raw_spin_unlock+0x75/0x190 [ 5848.281250] ? _raw_spin_unlock+0x12/0x30 [ 5848.282728] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5848.285939] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5848.288870] ll_atomic_open+0x36a/0xd70 [lustre] [ 5848.292471] lookup_open+0x356/0xb40 [ 5848.293640] do_last+0x313/0xfc0 [ 5848.303677] ? nd_jump_root+0xe5/0x160 [ 5848.309051] ? path_init+0x437/0x520 [ 5848.312020] path_openat+0xf7/0x500 [ 5848.314371] ? do_raw_spin_unlock+0x75/0x190 [ 5848.320327] do_filp_open+0x99/0x140 [ 5848.323705] ? getname_flags+0x6e/0x330 [ 5848.330558] ? __check_object_size+0xff/0x256 [ 5848.336122] ? do_raw_spin_unlock+0x75/0x190 [ 5848.339949] ? _raw_spin_unlock+0x12/0x30 [ 5848.342821] do_sys_openat2+0x2b4/0x410 [ 5848.347508] do_sys_open+0x73/0xa0 [ 5848.351271] __x64_sys_openat+0x24/0x30 [ 5848.355387] do_syscall_64+0xc1/0x440 [ 5848.358684] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5848.365840] RIP: 0033:0x7fdc0afbf332 [ 5848.374981] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5848.394010] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5848.396729] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5848.405427] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5848.414285] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5848.426748] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5848.436783] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5856.054592] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5856.060644] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5856.079539] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5856.083900] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5856.089122] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5856.092222] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5856.100479] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5856.103232] RAX: 00000000000069b8 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5856.106701] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5856.110466] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5856.113092] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5856.116822] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5856.121828] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5856.125164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5856.127833] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5856.131083] Call Trace: [ 5856.132192] [ 5856.136767] ? show_regs.cold.9+0x22/0x2f [ 5856.139694] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5856.142275] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5856.144421] ? watchdog+0x40/0x40 [ 5856.148850] ? hrtimer_interrupt+0x128/0x2c0 [ 5856.154219] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5856.156971] ? apic_timer_interrupt+0xf/0x20 [ 5856.161567] [ 5856.162843] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5856.165683] do_raw_spin_lock+0x177/0x190 [ 5856.167225] _raw_spin_lock+0x21/0x30 [ 5856.169147] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5856.172467] ? do_raw_spin_unlock+0x75/0x190 [ 5856.174729] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5856.177220] ? do_wait_intr+0xf0/0xf0 [ 5856.178860] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5856.182310] mdc_close+0x53b/0xd70 [mdc] [ 5856.184260] lmv_close+0x18d/0x490 [lmv] [ 5856.186203] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5856.189659] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5856.193376] ll_md_close+0x274/0x800 [lustre] [ 5856.195804] ll_file_release+0x128/0xc50 [lustre] [ 5856.197426] __fput+0x10d/0x440 [ 5856.198783] ____fput+0x12/0x20 [ 5856.200314] task_work_run+0xc6/0x110 [ 5856.202405] exit_to_usermode_loop+0x1dd/0x1f0 [ 5856.204259] do_syscall_64+0x3d6/0x440 [ 5856.205674] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5856.208125] RIP: 0033:0x7fdc0afbfcd8 [ 5856.210066] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5856.219600] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5856.222431] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5856.225398] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5856.228033] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5856.231199] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5856.234292] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5859.430012] rcu: INFO: rcu_sched self-detected stall on CPU [ 5859.437079] rcu: 0-...!: (2281348 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 5859.443368] (t=3840087 jiffies g=270761 q=3614) [ 5859.444430] rcu: rcu_sched kthread starved for 3835019 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 5859.447171] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5859.450099] rcu: RCU grace-period kthread stack dump: [ 5859.451739] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5859.456967] Call Trace: [ 5859.457783] __schedule+0x351/0xcb0 [ 5859.464066] schedule+0xc0/0x180 [ 5859.467927] schedule_timeout+0xb4/0x190 [ 5859.479826] ? do_raw_spin_unlock+0x75/0x190 [ 5859.483620] ? __next_timer_interrupt+0x160/0x160 [ 5859.487677] ? prepare_to_swait_event+0xf3/0x140 [ 5859.497297] rcu_gp_kthread+0x8dd/0x15b0 [ 5859.501366] ? force_qs_rnp+0x2c0/0x2c0 [ 5859.504459] kthread+0x1d1/0x200 [ 5859.506723] ? set_kthread_struct+0x70/0x70 [ 5859.508697] ret_from_fork+0x1f/0x30 [ 5859.510250] rcu: Stack dump where RCU GP kthread last ran: [ 5859.512647] Sending NMI from CPU 0 to CPUs 1: [ 5859.515708] NMI backtrace for cpu 1 [ 5859.515714] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5859.515716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5859.515718] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5859.515720] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5859.515722] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5859.515725] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 [ 5859.515726] RAX: 0000000000006105 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5859.515728] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5859.515729] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5859.515731] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5859.515732] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5859.515733] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5859.515734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5859.515735] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5859.515736] Call Trace: [ 5859.515737] [ 5859.515738] ? show_regs.cold.9+0x22/0x2f [ 5859.515740] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5859.515741] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5859.515742] ? nmi_handle+0xbd/0x210 [ 5859.515743] ? default_do_nmi+0x4b/0x1b0 [ 5859.515744] ? do_nmi+0x280/0x380 [ 5859.515745] ? end_repeat_nmi+0x16/0x1f [ 5859.515746] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5859.515747] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5859.515748] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5859.515749] [ 5859.515750] do_raw_spin_lock+0x177/0x190 [ 5859.515752] _raw_spin_lock+0x21/0x30 [ 5859.515753] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5859.515754] process_one_work+0x2c8/0x700 [ 5859.515755] worker_thread+0x296/0x6e0 [ 5859.515756] ? rescuer_thread+0x570/0x570 [ 5859.515757] kthread+0x1d1/0x200 [ 5859.515758] ? set_kthread_struct+0x70/0x70 [ 5859.515759] ret_from_fork+0x1f/0x30 [ 5859.517588] NMI backtrace for cpu 0 [ 5859.645949] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5859.650344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5859.654102] Call Trace: [ 5859.655444] [ 5859.656250] ? dump_stack+0xbb/0x10e [ 5859.657623] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5859.659973] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5859.661914] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5859.664062] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5859.666033] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5859.668088] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5859.671415] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5859.673963] ? update_process_times+0x6d/0xc0 [ 5859.675930] ? tick_sched_handle.isra.18+0x27/0x90 [ 5859.677761] ? tick_sched_timer+0x53/0xb0 [ 5859.679662] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5859.681619] ? tick_sched_handle.isra.18+0x90/0x90 [ 5859.683635] ? hrtimer_interrupt+0x128/0x2c0 [ 5859.684811] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5859.686743] ? apic_timer_interrupt+0xf/0x20 [ 5859.688675] [ 5859.689410] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5859.692735] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5859.695530] ? dump_stack+0xf6/0x10e [ 5859.696838] ? do_raw_spin_lock+0x177/0x190 [ 5859.698987] ? _raw_spin_lock+0x21/0x30 [ 5859.701600] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5859.709822] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5859.714237] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5859.717785] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5859.721197] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5859.723250] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5859.730819] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5859.736790] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5859.740080] ? do_wait_intr+0xf0/0xf0 [ 5859.741699] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5859.744319] ? kthread+0x1d1/0x200 [ 5859.746275] ? set_kthread_struct+0x70/0x70 [ 5859.751630] ? ret_from_fork+0x1f/0x30 [ 5873.120642] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3848s! [ 5873.133126] Showing busy workqueues and worker pools: [ 5873.134979] workqueue events: flags=0x0 [ 5873.150051] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5873.153455] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5873.164951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5873.172170] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5873.193333] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5873.197925] pending: kfree_rcu_monitor [ 5873.203369] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5873.207874] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5873.221876] workqueue events_long: flags=0x0 [ 5873.228517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5873.233199] pending: lu_objects_flush [obdclass] [ 5873.241789] workqueue events_unbound: flags=0x2 [ 5873.246319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5873.254923] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5873.262376] workqueue events_power_efficient: flags=0x80 [ 5873.264774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5873.268051] pending: neigh_periodic_work, check_lifetime [ 5873.270515] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5873.276762] pending: do_cache_clean, neigh_periodic_work [ 5873.280588] workqueue mm_percpu_wq: flags=0x8 [ 5873.283583] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5873.287345] pending: lru_add_drain_per_cpu [ 5873.291572] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5873.298479] pending: vmstat_update, lru_add_drain_per_cpu [ 5873.302358] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5873.305719] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5873.310369] workqueue writeback: flags=0x4a [ 5873.313016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5873.319341] pending: wb_workfn, wb_workfn [ 5873.321487] workqueue xprtiod: flags=0xa [ 5873.323111] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5873.325752] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5873.329748] workqueue ipv6_addrconf: flags=0x40008 [ 5873.332472] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5873.335998] pending: addrconf_verify_work [ 5873.341054] workqueue knbd0-recv: flags=0x1a [ 5873.343444] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5873.346118] in-flight: 655:recv_work [ 5873.348764] workqueue ptlrpc_pinger: flags=0x2 [ 5873.351097] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5873.354113] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5873.357200] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3848s workers=4 idle: 36396 38 11188 [ 5873.360978] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5863s workers=2 idle: 658 [ 5876.054789] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5876.060280] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5876.100565] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5876.107809] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5876.113228] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5876.119760] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5876.137678] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5876.150383] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5876.157357] RDX: 00000000000067c0 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5876.161252] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5876.165172] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5876.168030] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5876.172310] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5876.177384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5876.182780] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5876.189521] Call Trace: [ 5876.191790] [ 5876.193151] ? show_regs.cold.9+0x22/0x2f [ 5876.195333] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5876.199587] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5876.202389] ? watchdog+0x40/0x40 [ 5876.205777] ? hrtimer_interrupt+0x128/0x2c0 [ 5876.211215] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5876.215491] ? apic_timer_interrupt+0xf/0x20 [ 5876.219790] [ 5876.221027] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5876.226324] do_raw_spin_lock+0x177/0x190 [ 5876.228848] _raw_spin_lock+0x21/0x30 [ 5876.234283] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5876.242867] ? do_raw_spin_unlock+0x75/0x190 [ 5876.244807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5876.248350] ? do_wait_intr+0xf0/0xf0 [ 5876.250500] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5876.255633] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5876.257668] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5876.264469] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5876.269628] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5876.272545] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5876.275493] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5876.278482] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5876.279832] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5876.285188] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5876.288478] lmv_intent_open+0x338/0x1150 [lmv] [ 5876.294270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5876.301334] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5876.307281] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5876.313893] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5876.323233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5876.326477] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5876.340252] ? do_raw_spin_unlock+0x75/0x190 [ 5876.346514] ? _raw_spin_unlock+0x12/0x30 [ 5876.348023] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5876.351644] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5876.355854] ll_atomic_open+0x36a/0xd70 [lustre] [ 5876.359704] lookup_open+0x356/0xb40 [ 5876.361754] do_last+0x313/0xfc0 [ 5876.367395] ? nd_jump_root+0xe5/0x160 [ 5876.368423] ? path_init+0x437/0x520 [ 5876.370618] path_openat+0xf7/0x500 [ 5876.371930] ? do_raw_spin_unlock+0x75/0x190 [ 5876.374332] do_filp_open+0x99/0x140 [ 5876.376740] ? getname_flags+0x6e/0x330 [ 5876.379078] ? __check_object_size+0xff/0x256 [ 5876.384554] ? do_raw_spin_unlock+0x75/0x190 [ 5876.389524] ? _raw_spin_unlock+0x12/0x30 [ 5876.392070] do_sys_openat2+0x2b4/0x410 [ 5876.394803] do_sys_open+0x73/0xa0 [ 5876.398041] __x64_sys_openat+0x24/0x30 [ 5876.401042] do_syscall_64+0xc1/0x440 [ 5876.403732] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5876.406313] RIP: 0033:0x7fdc0afbf332 [ 5876.409386] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5876.420338] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5876.427797] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5876.432310] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5876.438266] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5876.442871] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5876.449790] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5884.051057] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 5884.053901] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5884.054099] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 5884.073245] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5884.073260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5884.073273] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5884.073721] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5884.073732] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5884.073736] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5884.073741] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5884.073743] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5884.073745] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5884.073748] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5884.073750] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5884.073753] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5884.073757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5884.073759] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5884.073769] Call Trace: [ 5884.073774] [ 5884.073780] ? show_regs.cold.9+0x22/0x2f [ 5884.073790] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5884.073796] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5884.073800] ? watchdog+0x40/0x40 [ 5884.073804] ? hrtimer_interrupt+0x128/0x2c0 [ 5884.073808] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5884.073815] ? apic_timer_interrupt+0xf/0x20 [ 5884.073820] [ 5884.073823] ? native_safe_halt+0x2e/0x50 [ 5884.073827] kvm_wait+0xcd/0xe0 [ 5884.073845] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5884.073853] do_raw_spin_lock+0x177/0x190 [ 5884.073858] _raw_spin_lock+0x21/0x30 [ 5884.073863] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5884.074097] process_one_work+0x2c8/0x700 [ 5884.074104] worker_thread+0x296/0x6e0 [ 5884.074108] ? rescuer_thread+0x570/0x570 [ 5884.074112] kthread+0x1d1/0x200 [ 5884.074117] ? set_kthread_struct+0x70/0x70 [ 5884.074120] ret_from_fork+0x1f/0x30 [ 5884.231059] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5884.280204] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5884.287743] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5884.294713] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5884.299493] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5884.336679] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5884.353256] RAX: 0000000000006cfb RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5884.363380] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5884.372029] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5884.379771] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5884.387129] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5884.398128] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5884.405208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5884.412073] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5884.420357] Call Trace: [ 5884.421674] [ 5884.422501] ? show_regs.cold.9+0x22/0x2f [ 5884.425307] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5884.431278] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5884.435875] ? watchdog+0x40/0x40 [ 5884.439245] ? hrtimer_interrupt+0x128/0x2c0 [ 5884.445283] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5884.449310] ? apic_timer_interrupt+0xf/0x20 [ 5884.456217] [ 5884.457487] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5884.459346] do_raw_spin_lock+0x177/0x190 [ 5884.460775] _raw_spin_lock+0x21/0x30 [ 5884.469860] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5884.477165] ? do_raw_spin_unlock+0x75/0x190 [ 5884.481924] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5884.484635] ? do_wait_intr+0xf0/0xf0 [ 5884.488805] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5884.493877] mdc_close+0x53b/0xd70 [mdc] [ 5884.499195] lmv_close+0x18d/0x490 [lmv] [ 5884.503261] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5884.516069] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5884.521073] ll_md_close+0x274/0x800 [lustre] [ 5884.534317] ll_file_release+0x128/0xc50 [lustre] [ 5884.539995] __fput+0x10d/0x440 [ 5884.544744] ____fput+0x12/0x20 [ 5884.545766] task_work_run+0xc6/0x110 [ 5884.548811] exit_to_usermode_loop+0x1dd/0x1f0 [ 5884.555888] do_syscall_64+0x3d6/0x440 [ 5884.564938] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5884.566681] RIP: 0033:0x7fdc0afbfcd8 [ 5884.576842] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5884.590442] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5884.599535] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5884.604739] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5884.611807] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5884.616877] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5884.626932] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5900.048045] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5900.055994] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5900.085688] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5900.094835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5900.099538] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5900.102742] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5900.113693] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5900.117790] RAX: 0000000000004a92 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5900.123854] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5900.127723] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5900.132239] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5900.135917] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5900.139540] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5900.144100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5900.147370] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5900.151158] Call Trace: [ 5900.153005] [ 5900.154175] ? show_regs.cold.9+0x22/0x2f [ 5900.156332] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5900.158639] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5900.161009] ? watchdog+0x40/0x40 [ 5900.163196] ? hrtimer_interrupt+0x128/0x2c0 [ 5900.165367] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5900.168224] ? apic_timer_interrupt+0xf/0x20 [ 5900.169735] [ 5900.171195] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5900.174075] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5900.176801] ? dump_stack+0xf6/0x10e [ 5900.179018] do_raw_spin_lock+0x177/0x190 [ 5900.181998] _raw_spin_lock+0x21/0x30 [ 5900.183687] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5900.186887] obd_mod_free+0x46/0x140 [ptlrpc] [ 5900.189374] mdc_commit_open+0x91/0xb0 [mdc] [ 5900.191350] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5900.194709] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5900.197491] after_reply+0xe2c/0x1780 [ptlrpc] [ 5900.199952] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5900.202410] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5900.205033] ? do_wait_intr+0xf0/0xf0 [ 5900.210227] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5900.213353] kthread+0x1d1/0x200 [ 5900.215000] ? set_kthread_struct+0x70/0x70 [ 5900.217018] ret_from_fork+0x1f/0x30 [ 5903.839280] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3878s! [ 5903.843328] Showing busy workqueues and worker pools: [ 5903.846457] workqueue events: flags=0x0 [ 5903.848537] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5903.853042] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5903.857679] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5903.860287] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5903.864653] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5903.867866] pending: kfree_rcu_monitor [ 5903.869457] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5903.872834] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5903.880682] workqueue events_long: flags=0x0 [ 5903.882834] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5903.886303] pending: lu_objects_flush [obdclass] [ 5903.889890] workqueue events_unbound: flags=0x2 [ 5903.891730] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5903.894777] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5903.900907] workqueue events_power_efficient: flags=0x80 [ 5903.903193] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5903.908863] pending: neigh_periodic_work, check_lifetime [ 5903.911292] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5903.914005] pending: do_cache_clean, neigh_periodic_work [ 5903.916989] workqueue mm_percpu_wq: flags=0x8 [ 5903.919472] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5903.923217] pending: lru_add_drain_per_cpu [ 5903.925273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5903.929213] pending: vmstat_update, lru_add_drain_per_cpu [ 5903.932330] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5903.936953] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5903.940244] workqueue writeback: flags=0x4a [ 5903.941946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5903.945151] pending: wb_workfn, wb_workfn [ 5903.948232] workqueue xprtiod: flags=0xa [ 5903.949762] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5903.952132] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5903.955690] workqueue ipv6_addrconf: flags=0x40008 [ 5903.958499] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5903.961429] pending: addrconf_verify_work [ 5903.963413] workqueue knbd0-recv: flags=0x1a [ 5903.965389] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5903.967932] in-flight: 655:recv_work [ 5903.969747] workqueue ptlrpc_pinger: flags=0x2 [ 5903.972348] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5903.975727] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5903.978187] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3879s workers=4 idle: 36396 38 11188 [ 5903.981883] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5894s workers=2 idle: 658 [ 5904.052017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5904.055812] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5904.104803] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5904.119702] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5904.133239] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5904.140121] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5904.157291] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5904.162049] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5904.164259] RDX: 0000000000006177 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5904.169749] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5904.175645] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5904.180908] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5904.185873] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5904.191174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5904.196808] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5904.200876] Call Trace: [ 5904.201877] [ 5904.202923] ? show_regs.cold.9+0x22/0x2f [ 5904.205044] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5904.206992] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5904.210049] ? watchdog+0x40/0x40 [ 5904.214530] ? hrtimer_interrupt+0x128/0x2c0 [ 5904.219740] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5904.222928] ? apic_timer_interrupt+0xf/0x20 [ 5904.225026] [ 5904.226416] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5904.229238] do_raw_spin_lock+0x177/0x190 [ 5904.231303] _raw_spin_lock+0x21/0x30 [ 5904.233047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5904.238378] ? do_raw_spin_unlock+0x75/0x190 [ 5904.240939] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5904.244017] ? do_wait_intr+0xf0/0xf0 [ 5904.246025] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5904.248990] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5904.252380] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5904.255690] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5904.262273] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5904.267032] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5904.276682] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5904.279593] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5904.282647] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5904.286151] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5904.288669] lmv_intent_open+0x338/0x1150 [lmv] [ 5904.294053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5904.301650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5904.305200] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5904.307297] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5904.311308] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5904.313310] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5904.318402] ? do_raw_spin_unlock+0x75/0x190 [ 5904.320676] ? _raw_spin_unlock+0x12/0x30 [ 5904.322710] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5904.326607] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5904.329972] ll_atomic_open+0x36a/0xd70 [lustre] [ 5904.341059] lookup_open+0x356/0xb40 [ 5904.346137] do_last+0x313/0xfc0 [ 5904.348125] ? nd_jump_root+0xe5/0x160 [ 5904.354607] ? path_init+0x437/0x520 [ 5904.358771] path_openat+0xf7/0x500 [ 5904.360025] ? do_raw_spin_unlock+0x75/0x190 [ 5904.362724] do_filp_open+0x99/0x140 [ 5904.364473] ? getname_flags+0x6e/0x330 [ 5904.367028] ? __check_object_size+0xff/0x256 [ 5904.368763] ? do_raw_spin_unlock+0x75/0x190 [ 5904.370041] ? _raw_spin_unlock+0x12/0x30 [ 5904.372961] do_sys_openat2+0x2b4/0x410 [ 5904.374653] do_sys_open+0x73/0xa0 [ 5904.376000] __x64_sys_openat+0x24/0x30 [ 5904.377253] do_syscall_64+0xc1/0x440 [ 5904.378227] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5904.379467] RIP: 0033:0x7fdc0afbf332 [ 5904.382118] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5904.392770] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5904.402538] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5904.406996] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5904.415084] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5904.417397] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5904.421510] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5912.052191] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 5912.058047] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5912.058222] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5912.075826] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5912.075840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5912.075849] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5912.076287] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5912.076298] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5912.076303] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5912.076307] RAX: 0000000000007184 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5912.076310] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5912.076312] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5912.076315] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5912.076317] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5912.076321] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5912.076324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5912.076326] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5912.076334] Call Trace: [ 5912.076339] [ 5912.076355] ? show_regs.cold.9+0x22/0x2f [ 5912.076365] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5912.076371] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5912.076376] ? watchdog+0x40/0x40 [ 5912.076380] ? hrtimer_interrupt+0x128/0x2c0 [ 5912.076384] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5912.076393] ? apic_timer_interrupt+0xf/0x20 [ 5912.076397] [ 5912.076404] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5912.076413] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5912.076421] do_raw_spin_lock+0x177/0x190 [ 5912.076426] _raw_spin_lock+0x21/0x30 [ 5912.076433] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5912.076670] process_one_work+0x2c8/0x700 [ 5912.076681] worker_thread+0x296/0x6e0 [ 5912.076685] ? rescuer_thread+0x570/0x570 [ 5912.076692] kthread+0x1d1/0x200 [ 5912.076696] ? set_kthread_struct+0x70/0x70 [ 5912.076700] ret_from_fork+0x1f/0x30 [ 5912.269626] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5912.324513] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5912.339551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5912.354767] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5912.358830] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5912.395744] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5912.407177] RAX: 0000000000006d85 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5912.412376] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5912.417845] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5912.421812] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5912.425917] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5912.432118] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5912.438318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5912.443128] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5912.446644] Call Trace: [ 5912.449100] [ 5912.450184] ? show_regs.cold.9+0x22/0x2f [ 5912.453747] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5912.458292] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5912.462262] ? watchdog+0x40/0x40 [ 5912.466343] ? hrtimer_interrupt+0x128/0x2c0 [ 5912.470215] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5912.473611] ? apic_timer_interrupt+0xf/0x20 [ 5912.476478] [ 5912.477681] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5912.480795] do_raw_spin_lock+0x177/0x190 [ 5912.483217] _raw_spin_lock+0x21/0x30 [ 5912.485395] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5912.488891] ? do_raw_spin_unlock+0x75/0x190 [ 5912.491632] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5912.495327] ? do_wait_intr+0xf0/0xf0 [ 5912.497615] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5912.501344] mdc_close+0x53b/0xd70 [mdc] [ 5912.504098] lmv_close+0x18d/0x490 [lmv] [ 5912.506311] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5912.510363] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5912.513272] ll_md_close+0x274/0x800 [lustre] [ 5912.517205] ll_file_release+0x128/0xc50 [lustre] [ 5912.520133] __fput+0x10d/0x440 [ 5912.522255] ____fput+0x12/0x20 [ 5912.523952] task_work_run+0xc6/0x110 [ 5912.526353] exit_to_usermode_loop+0x1dd/0x1f0 [ 5912.529487] do_syscall_64+0x3d6/0x440 [ 5912.532073] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5912.534697] RIP: 0033:0x7fdc0afbfcd8 [ 5912.536691] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5912.547572] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5912.551877] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5912.555643] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5912.559950] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5912.563297] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5912.567780] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5928.048953] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5928.056569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5928.102166] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5928.109573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5928.113625] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5928.116388] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5928.126294] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5928.130228] RAX: 0000000000003eb0 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5928.133357] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5928.140658] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5928.146688] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5928.150462] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5928.153758] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5928.158217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5928.161366] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5928.165017] Call Trace: [ 5928.166237] [ 5928.168324] ? show_regs.cold.9+0x22/0x2f [ 5928.170181] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5928.172163] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5928.173761] ? watchdog+0x40/0x40 [ 5928.176822] ? hrtimer_interrupt+0x128/0x2c0 [ 5928.178388] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5928.183077] ? apic_timer_interrupt+0xf/0x20 [ 5928.184933] [ 5928.185757] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5928.187851] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5928.189971] ? dump_stack+0xf6/0x10e [ 5928.191398] do_raw_spin_lock+0x177/0x190 [ 5928.193016] _raw_spin_lock+0x21/0x30 [ 5928.197919] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5928.200614] obd_mod_free+0x46/0x140 [ptlrpc] [ 5928.203929] mdc_commit_open+0x91/0xb0 [mdc] [ 5928.205489] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5928.211070] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5928.213492] after_reply+0xe2c/0x1780 [ptlrpc] [ 5928.218747] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5928.222183] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5928.225037] ? do_wait_intr+0xf0/0xf0 [ 5928.228691] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5928.233597] kthread+0x1d1/0x200 [ 5928.234622] ? set_kthread_struct+0x70/0x70 [ 5928.237314] ret_from_fork+0x1f/0x30 [ 5932.055797] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5932.061723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5932.098694] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5932.105880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5932.111689] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5932.118632] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5932.136926] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5932.140917] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5932.145110] RDX: 000000000000536d RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5932.155928] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5932.165546] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5932.170932] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5932.174446] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5932.181153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5932.186343] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5932.191561] Call Trace: [ 5932.193579] [ 5932.195719] ? show_regs.cold.9+0x22/0x2f [ 5932.200339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5932.204532] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5932.208082] ? watchdog+0x40/0x40 [ 5932.209295] ? hrtimer_interrupt+0x128/0x2c0 [ 5932.210591] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5932.214145] ? apic_timer_interrupt+0xf/0x20 [ 5932.215711] [ 5932.216367] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5932.219324] do_raw_spin_lock+0x177/0x190 [ 5932.222676] _raw_spin_lock+0x21/0x30 [ 5932.224196] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5932.228968] ? do_raw_spin_unlock+0x75/0x190 [ 5932.236880] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5932.242790] ? do_wait_intr+0xf0/0xf0 [ 5932.245304] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5932.249966] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5932.262799] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5932.267231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5932.271000] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5932.280928] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5932.285762] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5932.289531] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5932.295404] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5932.301040] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5932.305016] lmv_intent_open+0x338/0x1150 [lmv] [ 5932.309711] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5932.312900] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5932.318530] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5932.321867] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5932.327326] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5932.332029] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5932.337537] ? do_raw_spin_unlock+0x75/0x190 [ 5932.339925] ? _raw_spin_unlock+0x12/0x30 [ 5932.341447] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5932.345549] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5932.349277] ll_atomic_open+0x36a/0xd70 [lustre] [ 5932.352415] lookup_open+0x356/0xb40 [ 5932.354428] do_last+0x313/0xfc0 [ 5932.360307] ? nd_jump_root+0xe5/0x160 [ 5932.363081] ? path_init+0x437/0x520 [ 5932.366528] path_openat+0xf7/0x500 [ 5932.368043] ? do_raw_spin_unlock+0x75/0x190 [ 5932.370666] do_filp_open+0x99/0x140 [ 5932.372827] ? getname_flags+0x6e/0x330 [ 5932.374803] ? __check_object_size+0xff/0x256 [ 5932.377418] ? do_raw_spin_unlock+0x75/0x190 [ 5932.379413] ? _raw_spin_unlock+0x12/0x30 [ 5932.380849] do_sys_openat2+0x2b4/0x410 [ 5932.385264] do_sys_open+0x73/0xa0 [ 5932.387053] __x64_sys_openat+0x24/0x30 [ 5932.390135] do_syscall_64+0xc1/0x440 [ 5932.394538] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5932.398173] RIP: 0033:0x7fdc0afbf332 [ 5932.401387] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5932.411356] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5932.414084] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5932.423663] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5932.428645] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5932.431496] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5932.434029] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5934.047038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3909s! [ 5934.049847] Showing busy workqueues and worker pools: [ 5934.051518] workqueue events: flags=0x0 [ 5934.053845] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5934.058788] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5934.061085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5934.066476] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5934.073098] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5934.076189] pending: kfree_rcu_monitor [ 5934.078932] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5934.085179] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5934.094566] workqueue events_long: flags=0x0 [ 5934.096035] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5934.107149] pending: lu_objects_flush [obdclass] [ 5934.113438] workqueue events_unbound: flags=0x2 [ 5934.118182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5934.130334] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5934.135475] workqueue events_power_efficient: flags=0x80 [ 5934.142094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5934.154311] pending: neigh_periodic_work, check_lifetime [ 5934.156436] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5934.164021] pending: do_cache_clean, neigh_periodic_work [ 5934.169206] workqueue mm_percpu_wq: flags=0x8 [ 5934.176089] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5934.179705] pending: lru_add_drain_per_cpu [ 5934.184997] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5934.187303] pending: vmstat_update, lru_add_drain_per_cpu [ 5934.193725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5934.200373] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5934.207006] workqueue writeback: flags=0x4a [ 5934.210798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5934.217686] pending: wb_workfn, wb_workfn [ 5934.220409] workqueue xprtiod: flags=0xa [ 5934.230304] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5934.239692] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5934.250273] workqueue ipv6_addrconf: flags=0x40008 [ 5934.256312] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5934.265194] pending: addrconf_verify_work [ 5934.269014] workqueue knbd0-recv: flags=0x1a [ 5934.271469] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5934.279846] in-flight: 655:recv_work [ 5934.283872] workqueue ptlrpc_pinger: flags=0x2 [ 5934.286013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5934.288411] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5934.292865] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3909s workers=4 idle: 36396 38 11188 [ 5934.300906] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5924s workers=2 idle: 658 [ 5940.051021] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 5940.053711] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5940.061085] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5940.107680] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5940.107694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5940.107701] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5940.109736] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5940.109750] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5940.109755] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5940.109759] RAX: 000000000000580e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5940.109762] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5940.109764] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5940.109766] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5940.109769] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5940.109772] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5940.109775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5940.109778] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5940.112266] Call Trace: [ 5940.113998] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5940.115791] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5940.115799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5940.115802] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5940.115816] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5940.115820] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5940.115825] RAX: 000000000000103a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5940.115827] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5940.115829] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5940.115831] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5940.115834] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5940.115837] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5940.115840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5940.115843] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5940.115860] Call Trace: [ 5940.115866] [ 5940.115872] ? show_regs.cold.9+0x22/0x2f [ 5940.115881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5940.115887] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5940.115892] ? watchdog+0x40/0x40 [ 5940.115897] ? hrtimer_interrupt+0x128/0x2c0 [ 5940.115901] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5940.115909] ? apic_timer_interrupt+0xf/0x20 [ 5940.115914] [ 5940.115917] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5940.115923] do_raw_spin_lock+0x177/0x190 [ 5940.115928] _raw_spin_lock+0x21/0x30 [ 5940.115934] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5940.117362] ? do_raw_spin_unlock+0x75/0x190 [ 5940.117370] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5940.117880] ? do_wait_intr+0xf0/0xf0 [ 5940.117889] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5940.118478] mdc_close+0x53b/0xd70 [mdc] [ 5940.118518] lmv_close+0x18d/0x490 [lmv] [ 5940.118548] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5940.118661] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5940.135547] [ 5940.135560] ? show_regs.cold.9+0x22/0x2f [ 5940.135576] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5940.135583] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5940.135588] ? watchdog+0x40/0x40 [ 5940.135593] ? hrtimer_interrupt+0x128/0x2c0 [ 5940.135597] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5940.135605] ? apic_timer_interrupt+0xf/0x20 [ 5940.135610] [ 5940.135613] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5940.135620] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5940.135628] do_raw_spin_lock+0x177/0x190 [ 5940.135632] _raw_spin_lock+0x21/0x30 [ 5940.135637] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5940.144962] ll_md_close+0x274/0x800 [lustre] [ 5940.149737] process_one_work+0x2c8/0x700 [ 5940.149758] worker_thread+0x296/0x6e0 [ 5940.149763] ? rescuer_thread+0x570/0x570 [ 5940.149767] kthread+0x1d1/0x200 [ 5940.149772] ? set_kthread_struct+0x70/0x70 [ 5940.149776] ret_from_fork+0x1f/0x30 [ 5940.682097] ll_file_release+0x128/0xc50 [lustre] [ 5940.683777] __fput+0x10d/0x440 [ 5940.684754] ____fput+0x12/0x20 [ 5940.694003] task_work_run+0xc6/0x110 [ 5940.698377] exit_to_usermode_loop+0x1dd/0x1f0 [ 5940.706374] do_syscall_64+0x3d6/0x440 [ 5940.719560] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5940.729302] RIP: 0033:0x7fdc0afbfcd8 [ 5940.740241] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5940.767485] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5940.769777] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5940.788928] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5940.801258] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5940.807933] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5940.810078] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5960.048799] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5960.052346] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5960.055854] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5960.056033] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5960.056041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5960.056044] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5960.056059] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5960.056063] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5960.056067] RAX: 0000000000000708 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5960.056069] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5960.056072] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5960.056073] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5960.056076] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5960.056079] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5960.056081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5960.056083] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5960.056091] Call Trace: [ 5960.056095] [ 5960.056101] ? show_regs.cold.9+0x22/0x2f [ 5960.056109] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5960.056115] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5960.056119] ? watchdog+0x40/0x40 [ 5960.056123] ? hrtimer_interrupt+0x128/0x2c0 [ 5960.056127] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5960.056136] ? apic_timer_interrupt+0xf/0x20 [ 5960.056141] [ 5960.056144] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5960.056148] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5960.056152] ? dump_stack+0xf6/0x10e [ 5960.056158] do_raw_spin_lock+0x177/0x190 [ 5960.056164] _raw_spin_lock+0x21/0x30 [ 5960.056170] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5960.060807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5960.060909] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5960.060919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5960.060922] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5960.060938] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5960.060946] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5960.060950] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5960.060953] RDX: 00000000000012dc RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5960.060955] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5960.060957] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5960.060962] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5960.060966] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5960.060969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5960.060974] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5960.060981] Call Trace: [ 5960.060986] [ 5960.060995] ? show_regs.cold.9+0x22/0x2f [ 5960.061005] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5960.061011] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5960.061019] ? watchdog+0x40/0x40 [ 5960.061023] ? hrtimer_interrupt+0x128/0x2c0 [ 5960.061031] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5960.061039] ? apic_timer_interrupt+0xf/0x20 [ 5960.061047] [ 5960.061051] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5960.061056] do_raw_spin_lock+0x177/0x190 [ 5960.061064] _raw_spin_lock+0x21/0x30 [ 5960.061073] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5960.061778] ? do_raw_spin_unlock+0x75/0x190 [ 5960.061785] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5960.088708] obd_mod_free+0x46/0x140 [ptlrpc] [ 5960.090027] mdc_commit_open+0x91/0xb0 [mdc] [ 5960.090072] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5960.090558] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5960.090986] after_reply+0xe2c/0x1780 [ptlrpc] [ 5960.091353] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5960.091568] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5960.092027] ? do_wait_intr+0xf0/0xf0 [ 5960.092037] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5960.092420] kthread+0x1d1/0x200 [ 5960.092426] ? set_kthread_struct+0x70/0x70 [ 5960.092430] ret_from_fork+0x1f/0x30 [ 5960.327926] ? do_wait_intr+0xf0/0xf0 [ 5960.329934] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5960.333749] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5960.336541] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5960.339781] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5960.342484] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5960.345008] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5960.347232] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5960.349612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5960.351797] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5960.354891] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5960.358103] lmv_intent_open+0x338/0x1150 [lmv] [ 5960.360178] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5960.363981] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5960.366712] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5960.368978] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5960.371585] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5960.374955] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5960.377535] ? do_raw_spin_unlock+0x75/0x190 [ 5960.379732] ? _raw_spin_unlock+0x12/0x30 [ 5960.381956] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5960.385083] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5960.388151] ll_atomic_open+0x36a/0xd70 [lustre] [ 5960.391039] lookup_open+0x356/0xb40 [ 5960.392865] do_last+0x313/0xfc0 [ 5960.394234] ? nd_jump_root+0xe5/0x160 [ 5960.396104] ? path_init+0x437/0x520 [ 5960.397963] path_openat+0xf7/0x500 [ 5960.399962] ? do_raw_spin_unlock+0x75/0x190 [ 5960.401907] do_filp_open+0x99/0x140 [ 5960.403829] ? getname_flags+0x6e/0x330 [ 5960.405733] ? __check_object_size+0xff/0x256 [ 5960.408594] ? do_raw_spin_unlock+0x75/0x190 [ 5960.410762] ? _raw_spin_unlock+0x12/0x30 [ 5960.413189] do_sys_openat2+0x2b4/0x410 [ 5960.415056] do_sys_open+0x73/0xa0 [ 5960.416891] __x64_sys_openat+0x24/0x30 [ 5960.418891] do_syscall_64+0xc1/0x440 [ 5960.420798] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5960.424067] RIP: 0033:0x7fdc0afbf332 [ 5960.426364] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5960.435987] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5960.440148] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5960.446441] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5960.458812] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5960.463426] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5960.467728] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5964.767874] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3939s! [ 5964.776147] Showing busy workqueues and worker pools: [ 5964.780161] workqueue events: flags=0x0 [ 5964.785879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5964.790970] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5964.801806] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5964.806801] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5964.813530] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5964.827375] pending: kfree_rcu_monitor [ 5964.831694] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5964.835204] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5964.852117] workqueue events_long: flags=0x0 [ 5964.854344] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5964.863264] pending: lu_objects_flush [obdclass] [ 5964.865113] workqueue events_unbound: flags=0x2 [ 5964.872764] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5964.876818] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5964.883849] workqueue events_power_efficient: flags=0x80 [ 5964.887439] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5964.895606] pending: neigh_periodic_work, check_lifetime [ 5964.899518] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5964.907300] pending: do_cache_clean, neigh_periodic_work [ 5964.912345] workqueue mm_percpu_wq: flags=0x8 [ 5964.913959] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5964.923560] pending: lru_add_drain_per_cpu [ 5964.931459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5964.936309] pending: vmstat_update, lru_add_drain_per_cpu [ 5964.939655] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5964.943774] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5964.949912] workqueue writeback: flags=0x4a [ 5964.952719] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5964.958076] pending: wb_workfn, wb_workfn [ 5964.960298] workqueue xprtiod: flags=0xa [ 5964.962060] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5964.965576] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5964.973207] workqueue ipv6_addrconf: flags=0x40008 [ 5964.977113] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5964.985156] pending: addrconf_verify_work [ 5964.990740] workqueue knbd0-recv: flags=0x1a [ 5964.992182] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5964.995872] in-flight: 655:recv_work [ 5964.999228] workqueue ptlrpc_pinger: flags=0x2 [ 5965.001655] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5965.012232] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5965.016150] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3940s workers=4 idle: 36396 38 11188 [ 5965.026536] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5955s workers=2 idle: 658 [ 5968.053586] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 5968.055636] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5968.059560] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5968.059641] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5968.059649] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5968.059656] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5968.059932] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5968.059943] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5968.059947] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5968.059952] RAX: 0000000000007e85 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5968.059955] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5968.059957] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5968.059959] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5968.059961] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5968.059964] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5968.059967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5968.059970] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5968.059979] Call Trace: [ 5968.059985] [ 5968.059991] ? show_regs.cold.9+0x22/0x2f [ 5968.060001] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5968.060008] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5968.060013] ? watchdog+0x40/0x40 [ 5968.060017] ? hrtimer_interrupt+0x128/0x2c0 [ 5968.060022] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5968.060031] ? apic_timer_interrupt+0xf/0x20 [ 5968.060036] [ 5968.060039] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5968.060044] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5968.060050] do_raw_spin_lock+0x177/0x190 [ 5968.060055] _raw_spin_lock+0x21/0x30 [ 5968.060060] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5968.060424] process_one_work+0x2c8/0x700 [ 5968.060432] worker_thread+0x296/0x6e0 [ 5968.060436] ? rescuer_thread+0x570/0x570 [ 5968.060439] kthread+0x1d1/0x200 [ 5968.060443] ? set_kthread_struct+0x70/0x70 [ 5968.060446] ret_from_fork+0x1f/0x30 [ 5968.237670] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5968.260203] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5968.266187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5968.270335] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5968.274186] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5968.287328] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5968.294734] RAX: 0000000000007908 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5968.299344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5968.309152] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5968.317854] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5968.332727] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5968.339304] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5968.343383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5968.352264] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5968.358692] Call Trace: [ 5968.361234] [ 5968.361898] ? show_regs.cold.9+0x22/0x2f [ 5968.367848] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5968.374006] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5968.377552] ? watchdog+0x40/0x40 [ 5968.380250] ? hrtimer_interrupt+0x128/0x2c0 [ 5968.385551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5968.390587] ? apic_timer_interrupt+0xf/0x20 [ 5968.397197] [ 5968.397905] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5968.408549] do_raw_spin_lock+0x177/0x190 [ 5968.410328] _raw_spin_lock+0x21/0x30 [ 5968.411923] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5968.426934] ? do_raw_spin_unlock+0x75/0x190 [ 5968.430883] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5968.438790] ? do_wait_intr+0xf0/0xf0 [ 5968.440127] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5968.443869] mdc_close+0x53b/0xd70 [mdc] [ 5968.450595] lmv_close+0x18d/0x490 [lmv] [ 5968.452177] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5968.455726] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5968.457426] ll_md_close+0x274/0x800 [lustre] [ 5968.465627] ll_file_release+0x128/0xc50 [lustre] [ 5968.470792] __fput+0x10d/0x440 [ 5968.474740] ____fput+0x12/0x20 [ 5968.478192] task_work_run+0xc6/0x110 [ 5968.479295] exit_to_usermode_loop+0x1dd/0x1f0 [ 5968.484642] do_syscall_64+0x3d6/0x440 [ 5968.485896] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5968.488919] RIP: 0033:0x7fdc0afbfcd8 [ 5968.490051] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5968.503013] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5968.506419] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5968.509378] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5968.515576] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5968.519071] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5968.524552] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5988.052379] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 5988.054926] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5988.108903] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5988.121426] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5988.141571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5988.147999] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5988.169718] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5988.177462] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 5988.184529] RDX: 0000000000004b75 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 5988.193090] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 5988.196424] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 5988.198432] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5988.200622] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 5988.202779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5988.204846] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 5988.207464] Call Trace: [ 5988.208209] [ 5988.208780] ? show_regs.cold.9+0x22/0x2f [ 5988.210023] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5988.211795] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5988.213797] ? watchdog+0x40/0x40 [ 5988.214891] ? hrtimer_interrupt+0x128/0x2c0 [ 5988.216272] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5988.217484] ? apic_timer_interrupt+0xf/0x20 [ 5988.218505] [ 5988.219527] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5988.220953] do_raw_spin_lock+0x177/0x190 [ 5988.221916] _raw_spin_lock+0x21/0x30 [ 5988.222976] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5988.224785] ? do_raw_spin_unlock+0x75/0x190 [ 5988.226189] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5988.227670] ? do_wait_intr+0xf0/0xf0 [ 5988.228847] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5988.230275] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5988.231940] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5988.233959] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.235722] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5988.237698] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5988.239083] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5988.240770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.242212] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5988.244114] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5988.245987] lmv_intent_open+0x338/0x1150 [lmv] [ 5988.247342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.249014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.250520] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5988.251924] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5988.253230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.254966] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5988.256495] ? do_raw_spin_unlock+0x75/0x190 [ 5988.257786] ? _raw_spin_unlock+0x12/0x30 [ 5988.258907] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5988.260824] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5988.262038] ll_atomic_open+0x36a/0xd70 [lustre] [ 5988.263628] lookup_open+0x356/0xb40 [ 5988.264753] do_last+0x313/0xfc0 [ 5988.265729] ? nd_jump_root+0xe5/0x160 [ 5988.267215] ? path_init+0x437/0x520 [ 5988.268476] path_openat+0xf7/0x500 [ 5988.269483] ? do_raw_spin_unlock+0x75/0x190 [ 5988.270910] do_filp_open+0x99/0x140 [ 5988.272023] ? getname_flags+0x6e/0x330 [ 5988.272994] ? __check_object_size+0xff/0x256 [ 5988.274612] ? do_raw_spin_unlock+0x75/0x190 [ 5988.275773] ? _raw_spin_unlock+0x12/0x30 [ 5988.276954] do_sys_openat2+0x2b4/0x410 [ 5988.278168] do_sys_open+0x73/0xa0 [ 5988.279113] __x64_sys_openat+0x24/0x30 [ 5988.280317] do_syscall_64+0xc1/0x440 [ 5988.281485] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5988.282926] RIP: 0033:0x7fdc0afbf332 [ 5988.283867] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5988.289308] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5988.291796] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 5988.293891] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 5988.295908] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 5988.297933] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 5988.299896] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 5992.048012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 5992.052895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5992.081040] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5992.087200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5992.092572] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5992.096316] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5992.106898] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5992.111120] RAX: 00000000000057ac RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5992.115077] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 5992.119149] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 5992.122730] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 5992.126631] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 5992.130394] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 5992.135215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5992.138281] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 5992.142185] Call Trace: [ 5992.143305] [ 5992.144496] ? show_regs.cold.9+0x22/0x2f [ 5992.146727] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5992.149444] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5992.152110] ? watchdog+0x40/0x40 [ 5992.153688] ? hrtimer_interrupt+0x128/0x2c0 [ 5992.156271] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5992.159041] ? apic_timer_interrupt+0xf/0x20 [ 5992.161302] [ 5992.162161] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5992.164644] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5992.167529] ? dump_stack+0xf6/0x10e [ 5992.169413] do_raw_spin_lock+0x177/0x190 [ 5992.171604] _raw_spin_lock+0x21/0x30 [ 5992.173583] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5992.175430] obd_mod_free+0x46/0x140 [ptlrpc] [ 5992.176871] mdc_commit_open+0x91/0xb0 [mdc] [ 5992.179188] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5992.181338] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5992.183480] after_reply+0xe2c/0x1780 [ptlrpc] [ 5992.185868] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5992.188223] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5992.189964] ? do_wait_intr+0xf0/0xf0 [ 5992.191274] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5992.193909] kthread+0x1d1/0x200 [ 5992.195146] ? set_kthread_struct+0x70/0x70 [ 5992.196487] ret_from_fork+0x1f/0x30 [ 5995.488636] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3970s! [ 5995.499453] Showing busy workqueues and worker pools: [ 5995.508251] workqueue events: flags=0x0 [ 5995.510097] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5995.516320] pending: cpu_latency_work [ptlrpc], push_to_pool [ 5995.520071] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5995.535822] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 5995.548139] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5995.553362] pending: kfree_rcu_monitor [ 5995.558240] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5995.568353] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 5995.577210] workqueue events_long: flags=0x0 [ 5995.579221] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5995.587840] pending: lu_objects_flush [obdclass] [ 5995.593225] workqueue events_unbound: flags=0x2 [ 5995.599350] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5995.602553] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5995.612332] workqueue events_power_efficient: flags=0x80 [ 5995.618989] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5995.623945] pending: neigh_periodic_work, check_lifetime [ 5995.627460] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5995.633742] pending: do_cache_clean, neigh_periodic_work [ 5995.638903] workqueue mm_percpu_wq: flags=0x8 [ 5995.641053] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5995.646663] pending: lru_add_drain_per_cpu [ 5995.648720] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5995.654175] pending: vmstat_update, lru_add_drain_per_cpu [ 5995.659862] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5995.666342] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5995.670318] workqueue writeback: flags=0x4a [ 5995.674830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5995.678860] pending: wb_workfn, wb_workfn [ 5995.683061] workqueue xprtiod: flags=0xa [ 5995.689075] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5995.698024] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5995.700661] workqueue ipv6_addrconf: flags=0x40008 [ 5995.709374] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5995.717438] pending: addrconf_verify_work [ 5995.727512] workqueue knbd0-recv: flags=0x1a [ 5995.733146] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5995.736958] in-flight: 655:recv_work [ 5995.738415] workqueue ptlrpc_pinger: flags=0x2 [ 5995.739850] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5995.745986] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 5995.752593] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3970s workers=4 idle: 36396 38 11188 [ 5995.765246] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5986s workers=2 idle: 658 [ 5996.051032] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 5996.054032] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 5996.056791] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5996.056867] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5996.056882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5996.056889] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5996.061776] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5996.061862] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5996.061878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5996.061881] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5996.061894] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5996.061899] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5996.061903] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5996.061906] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 5996.061909] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5996.062033] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 5996.062036] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 5996.062039] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 5996.062043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5996.062045] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 5996.062054] Call Trace: [ 5996.062060] [ 5996.062065] ? show_regs.cold.9+0x22/0x2f [ 5996.062075] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5996.062081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5996.062086] ? watchdog+0x40/0x40 [ 5996.062090] ? hrtimer_interrupt+0x128/0x2c0 [ 5996.062095] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5996.062102] ? apic_timer_interrupt+0xf/0x20 [ 5996.062106] [ 5996.062109] ? native_safe_halt+0x2e/0x50 [ 5996.062113] kvm_wait+0xcd/0xe0 [ 5996.062120] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5996.062129] do_raw_spin_lock+0x177/0x190 [ 5996.062134] _raw_spin_lock+0x21/0x30 [ 5996.062138] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5996.062470] ? do_raw_spin_unlock+0x75/0x190 [ 5996.062476] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5996.062697] ? do_wait_intr+0xf0/0xf0 [ 5996.062704] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5996.062962] mdc_close+0x53b/0xd70 [mdc] [ 5996.063001] lmv_close+0x18d/0x490 [lmv] [ 5996.063033] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5996.063140] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5996.063225] ll_md_close+0x274/0x800 [lustre] [ 5996.063322] ll_file_release+0x128/0xc50 [lustre] [ 5996.063418] __fput+0x10d/0x440 [ 5996.063427] ____fput+0x12/0x20 [ 5996.063432] task_work_run+0xc6/0x110 [ 5996.063438] exit_to_usermode_loop+0x1dd/0x1f0 [ 5996.063445] do_syscall_64+0x3d6/0x440 [ 5996.063450] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5996.063457] RIP: 0033:0x7fdc0afbfcd8 [ 5996.063609] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5996.063613] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5996.063618] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 5996.063620] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 5996.063623] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 5996.063625] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 5996.063628] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5996.320391] [ 5996.321049] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5996.326062] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5996.346027] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5996.351057] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 5996.354919] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 5996.360090] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5996.367277] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 5996.373303] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 5996.383237] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 5996.389384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5996.394240] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 5996.401909] Call Trace: [ 5996.403429] [ 5996.406666] ? show_regs.cold.9+0x22/0x2f [ 5996.411142] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5996.413958] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5996.416455] ? watchdog+0x40/0x40 [ 5996.421650] ? hrtimer_interrupt+0x128/0x2c0 [ 5996.424364] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5996.427257] ? apic_timer_interrupt+0xf/0x20 [ 5996.428980] [ 5996.430325] ? native_safe_halt+0x2e/0x50 [ 5996.431930] kvm_wait+0xcd/0xe0 [ 5996.436489] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5996.438291] do_raw_spin_lock+0x177/0x190 [ 5996.441370] _raw_spin_lock+0x21/0x30 [ 5996.443747] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5996.452104] process_one_work+0x2c8/0x700 [ 5996.457659] worker_thread+0x296/0x6e0 [ 5996.459956] ? rescuer_thread+0x570/0x570 [ 5996.462102] kthread+0x1d1/0x200 [ 5996.464669] ? set_kthread_struct+0x70/0x70 [ 5996.469557] ret_from_fork+0x1f/0x30 [ 6016.052171] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6016.057225] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6016.089259] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6016.103670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6016.117850] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6016.120253] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6016.127484] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6016.134221] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6016.140056] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6016.142608] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6016.148254] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6016.152056] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6016.156428] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6016.160625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6016.163939] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6016.167282] Call Trace: [ 6016.168592] [ 6016.169635] ? show_regs.cold.9+0x22/0x2f [ 6016.171881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6016.174284] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6016.176397] ? watchdog+0x40/0x40 [ 6016.178074] ? hrtimer_interrupt+0x128/0x2c0 [ 6016.180287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6016.182667] ? apic_timer_interrupt+0xf/0x20 [ 6016.184794] [ 6016.185730] ? native_safe_halt+0x2e/0x50 [ 6016.188516] kvm_wait+0xcd/0xe0 [ 6016.189963] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6016.192451] do_raw_spin_lock+0x177/0x190 [ 6016.194540] _raw_spin_lock+0x21/0x30 [ 6016.196432] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6016.199251] ? do_raw_spin_unlock+0x75/0x190 [ 6016.201275] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6016.211579] ? do_wait_intr+0xf0/0xf0 [ 6016.214670] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6016.218311] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6016.221259] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6016.224499] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.227592] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6016.230609] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6016.234102] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6016.236912] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.240230] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6016.244071] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6016.248376] lmv_intent_open+0x338/0x1150 [lmv] [ 6016.254423] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.259824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.264162] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6016.265785] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6016.272690] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.277719] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6016.280542] ? do_raw_spin_unlock+0x75/0x190 [ 6016.282562] ? _raw_spin_unlock+0x12/0x30 [ 6016.284729] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6016.287386] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6016.290438] ll_atomic_open+0x36a/0xd70 [lustre] [ 6016.292671] lookup_open+0x356/0xb40 [ 6016.294324] do_last+0x313/0xfc0 [ 6016.295722] ? nd_jump_root+0xe5/0x160 [ 6016.297497] ? path_init+0x437/0x520 [ 6016.299107] path_openat+0xf7/0x500 [ 6016.300716] ? do_raw_spin_unlock+0x75/0x190 [ 6016.302619] do_filp_open+0x99/0x140 [ 6016.304426] ? getname_flags+0x6e/0x330 [ 6016.306267] ? __check_object_size+0xff/0x256 [ 6016.308254] ? do_raw_spin_unlock+0x75/0x190 [ 6016.310951] ? _raw_spin_unlock+0x12/0x30 [ 6016.312858] do_sys_openat2+0x2b4/0x410 [ 6016.314667] do_sys_open+0x73/0xa0 [ 6016.316445] __x64_sys_openat+0x24/0x30 [ 6016.318402] do_syscall_64+0xc1/0x440 [ 6016.320479] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6016.323043] RIP: 0033:0x7fdc0afbf332 [ 6016.324663] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6016.333816] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6016.337302] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6016.340865] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6016.344451] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6016.347933] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6016.351249] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6020.048262] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6020.053485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6020.072467] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6020.079406] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6020.088022] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6020.091499] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6020.102708] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6020.105340] RAX: 0000000000007f83 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6020.115682] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6020.126698] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6020.137647] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6020.145065] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6020.157734] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6020.163344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6020.167223] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6020.170446] Call Trace: [ 6020.172725] [ 6020.174530] ? show_regs.cold.9+0x22/0x2f [ 6020.176128] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6020.179462] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6020.182554] ? watchdog+0x40/0x40 [ 6020.184394] ? hrtimer_interrupt+0x128/0x2c0 [ 6020.186798] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6020.189347] ? apic_timer_interrupt+0xf/0x20 [ 6020.191752] [ 6020.192687] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6020.197847] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6020.202897] ? dump_stack+0xf6/0x10e [ 6020.205763] do_raw_spin_lock+0x177/0x190 [ 6020.211995] _raw_spin_lock+0x21/0x30 [ 6020.214652] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6020.217890] obd_mod_free+0x46/0x140 [ptlrpc] [ 6020.221090] mdc_commit_open+0x91/0xb0 [mdc] [ 6020.222712] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6020.225749] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6020.229393] after_reply+0xe2c/0x1780 [ptlrpc] [ 6020.232230] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6020.235191] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6020.237487] ? do_wait_intr+0xf0/0xf0 [ 6020.239028] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6020.241290] kthread+0x1d1/0x200 [ 6020.242663] ? set_kthread_struct+0x70/0x70 [ 6020.244274] ret_from_fork+0x1f/0x30 [ 6024.054320] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 6024.055350] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6024.057459] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6024.057541] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6024.057549] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6024.057552] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6024.057566] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6024.057570] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6024.057575] RAX: 00000000000074c4 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6024.057577] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6024.057579] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6024.057581] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6024.057583] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6024.057586] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6024.057589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6024.057592] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6024.057599] Call Trace: [ 6024.057605] [ 6024.057611] ? show_regs.cold.9+0x22/0x2f [ 6024.057621] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6024.057628] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6024.057632] ? watchdog+0x40/0x40 [ 6024.057637] ? hrtimer_interrupt+0x128/0x2c0 [ 6024.057640] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6024.057648] ? apic_timer_interrupt+0xf/0x20 [ 6024.057653] [ 6024.057656] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6024.057662] do_raw_spin_lock+0x177/0x190 [ 6024.057668] _raw_spin_lock+0x21/0x30 [ 6024.057672] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6024.059338] ? do_raw_spin_unlock+0x75/0x190 [ 6024.059347] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6024.059843] ? do_wait_intr+0xf0/0xf0 [ 6024.059851] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6024.060310] mdc_close+0x53b/0xd70 [mdc] [ 6024.060347] lmv_close+0x18d/0x490 [lmv] [ 6024.060377] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6024.060509] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6024.060596] ll_md_close+0x274/0x800 [lustre] [ 6024.060680] ll_file_release+0x128/0xc50 [lustre] [ 6024.060883] __fput+0x10d/0x440 [ 6024.060893] ____fput+0x12/0x20 [ 6024.060897] task_work_run+0xc6/0x110 [ 6024.060904] exit_to_usermode_loop+0x1dd/0x1f0 [ 6024.060911] do_syscall_64+0x3d6/0x440 [ 6024.060915] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6024.060924] RIP: 0033:0x7fdc0afbfcd8 [ 6024.060932] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6024.060936] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6024.060941] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6024.060943] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6024.060945] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6024.060948] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6024.060950] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6024.338093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6024.380772] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6024.391153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6024.402368] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6024.413162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6024.418678] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6024.434955] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6024.442637] RAX: 0000000000007e55 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6024.448891] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6024.455453] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6024.459536] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6024.464867] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6024.471288] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6024.475630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6024.481333] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6024.486845] Call Trace: [ 6024.487701] [ 6024.488258] ? show_regs.cold.9+0x22/0x2f [ 6024.490795] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6024.495082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6024.498681] ? watchdog+0x40/0x40 [ 6024.501848] ? hrtimer_interrupt+0x128/0x2c0 [ 6024.504945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6024.508966] ? apic_timer_interrupt+0xf/0x20 [ 6024.512414] [ 6024.514358] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6024.520316] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6024.524761] do_raw_spin_lock+0x177/0x190 [ 6024.530092] _raw_spin_lock+0x21/0x30 [ 6024.531293] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6024.540637] process_one_work+0x2c8/0x700 [ 6024.542071] worker_thread+0x296/0x6e0 [ 6024.546421] ? rescuer_thread+0x570/0x570 [ 6024.554812] kthread+0x1d1/0x200 [ 6024.562072] ? set_kthread_struct+0x70/0x70 [ 6024.564204] ret_from_fork+0x1f/0x30 [ 6026.207416] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4001s! [ 6026.212014] Showing busy workqueues and worker pools: [ 6026.214642] workqueue events: flags=0x0 [ 6026.215957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6026.220185] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6026.223653] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6026.228123] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6026.231635] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6026.233742] pending: kfree_rcu_monitor [ 6026.235985] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6026.238811] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6026.244975] workqueue events_long: flags=0x0 [ 6026.246434] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6026.248828] pending: lu_objects_flush [obdclass] [ 6026.250968] workqueue events_unbound: flags=0x2 [ 6026.252768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6026.257592] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6026.261303] workqueue events_power_efficient: flags=0x80 [ 6026.263578] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6026.266618] pending: neigh_periodic_work, check_lifetime [ 6026.268722] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6026.271734] pending: do_cache_clean, neigh_periodic_work [ 6026.273558] workqueue mm_percpu_wq: flags=0x8 [ 6026.275124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6026.278001] pending: lru_add_drain_per_cpu [ 6026.281739] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6026.284912] pending: vmstat_update, lru_add_drain_per_cpu [ 6026.286884] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6026.289339] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6026.292557] workqueue writeback: flags=0x4a [ 6026.294059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6026.296000] pending: wb_workfn, wb_workfn [ 6026.299114] workqueue xprtiod: flags=0xa [ 6026.301294] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6026.303883] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6026.306806] workqueue ipv6_addrconf: flags=0x40008 [ 6026.309922] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6026.312093] pending: addrconf_verify_work [ 6026.313764] workqueue knbd0-recv: flags=0x1a [ 6026.315263] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6026.319193] in-flight: 655:recv_work [ 6026.320701] workqueue ptlrpc_pinger: flags=0x2 [ 6026.322403] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6026.324602] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6026.326657] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4001s workers=4 idle: 36396 38 11188 [ 6026.330491] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6016s workers=2 idle: 658 [ 6039.433054] rcu: INFO: rcu_sched self-detected stall on CPU [ 6039.437086] rcu: 0-...!: (2399996 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 6039.442251] (t=4020090 jiffies g=270761 q=3631) [ 6039.444370] rcu: rcu_sched kthread starved for 4015022 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 6039.448580] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6039.452422] rcu: RCU grace-period kthread stack dump: [ 6039.454757] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6039.458352] Call Trace: [ 6039.464411] __schedule+0x351/0xcb0 [ 6039.466695] schedule+0xc0/0x180 [ 6039.468127] schedule_timeout+0xb4/0x190 [ 6039.470463] ? do_raw_spin_unlock+0x75/0x190 [ 6039.472432] ? __next_timer_interrupt+0x160/0x160 [ 6039.474918] ? prepare_to_swait_event+0xf3/0x140 [ 6039.477459] rcu_gp_kthread+0x8dd/0x15b0 [ 6039.479130] ? force_qs_rnp+0x2c0/0x2c0 [ 6039.480789] kthread+0x1d1/0x200 [ 6039.482296] ? set_kthread_struct+0x70/0x70 [ 6039.484983] ret_from_fork+0x1f/0x30 [ 6039.486479] rcu: Stack dump where RCU GP kthread last ran: [ 6039.487861] NMI backtrace for cpu 0 [ 6039.488653] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6039.494072] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6039.496535] Call Trace: [ 6039.497654] [ 6039.498299] ? dump_stack+0xbb/0x10e [ 6039.499865] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6039.501586] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6039.503562] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6039.505572] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6039.511645] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6039.514371] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 6039.518581] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 6039.521745] ? update_process_times+0x6d/0xc0 [ 6039.522772] ? tick_sched_handle.isra.18+0x27/0x90 [ 6039.523980] ? tick_sched_timer+0x53/0xb0 [ 6039.528632] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6039.533220] ? tick_sched_handle.isra.18+0x90/0x90 [ 6039.536588] ? hrtimer_interrupt+0x128/0x2c0 [ 6039.537943] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6039.540078] ? apic_timer_interrupt+0xf/0x20 [ 6039.541872] [ 6039.544646] ? native_safe_halt+0x2e/0x50 [ 6039.551478] ? kvm_wait+0xcd/0xe0 [ 6039.552690] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6039.556080] ? dump_stack+0xf6/0x10e [ 6039.557577] ? do_raw_spin_lock+0x177/0x190 [ 6039.563091] ? _raw_spin_lock+0x21/0x30 [ 6039.567172] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6039.570958] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6039.574238] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6039.576517] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6039.579501] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6039.585218] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6039.589432] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6039.592447] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6039.595496] ? do_wait_intr+0xf0/0xf0 [ 6039.596693] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6039.599501] ? kthread+0x1d1/0x200 [ 6039.601541] ? set_kthread_struct+0x70/0x70 [ 6039.603369] ? ret_from_fork+0x1f/0x30 [ 6039.607476] NMI backtrace for cpu 0 [ 6039.609184] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6039.615619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6039.618669] Call Trace: [ 6039.620328] [ 6039.621058] ? dump_stack+0xbb/0x10e [ 6039.622050] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6039.623598] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6039.625984] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6039.627866] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6039.630178] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6039.633130] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6039.634867] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6039.636475] ? update_process_times+0x6d/0xc0 [ 6039.638250] ? tick_sched_handle.isra.18+0x27/0x90 [ 6039.640201] ? tick_sched_timer+0x53/0xb0 [ 6039.641861] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6039.643667] ? tick_sched_handle.isra.18+0x90/0x90 [ 6039.645986] ? hrtimer_interrupt+0x128/0x2c0 [ 6039.649933] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6039.654089] ? apic_timer_interrupt+0xf/0x20 [ 6039.655874] [ 6039.656400] ? native_safe_halt+0x2e/0x50 [ 6039.658311] ? kvm_wait+0xcd/0xe0 [ 6039.659536] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6039.661792] ? dump_stack+0xf6/0x10e [ 6039.662901] ? do_raw_spin_lock+0x177/0x190 [ 6039.664872] ? _raw_spin_lock+0x21/0x30 [ 6039.666852] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6039.668934] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6039.670934] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6039.672698] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6039.674969] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6039.677221] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6039.679134] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6039.681183] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6039.682995] ? do_wait_intr+0xf0/0xf0 [ 6039.684460] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6039.686798] ? kthread+0x1d1/0x200 [ 6039.688032] ? set_kthread_struct+0x70/0x70 [ 6039.689914] ? ret_from_fork+0x1f/0x30 [ 6044.052085] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6044.055677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6044.069823] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6044.073615] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6044.076810] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6044.078951] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6044.085428] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6044.088080] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6044.090235] RDX: 00000000000060a9 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6044.092699] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6044.094810] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6044.097056] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6044.099198] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6044.101359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6044.102933] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6044.105115] Call Trace: [ 6044.105787] [ 6044.106388] ? show_regs.cold.9+0x22/0x2f [ 6044.107380] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6044.108582] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6044.109984] ? watchdog+0x40/0x40 [ 6044.110969] ? hrtimer_interrupt+0x128/0x2c0 [ 6044.112162] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6044.113393] ? apic_timer_interrupt+0xf/0x20 [ 6044.114537] [ 6044.115155] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6044.116872] do_raw_spin_lock+0x177/0x190 [ 6044.118219] _raw_spin_lock+0x21/0x30 [ 6044.119462] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6044.121601] ? do_raw_spin_unlock+0x75/0x190 [ 6044.123127] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6044.125291] ? do_wait_intr+0xf0/0xf0 [ 6044.127162] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6044.129227] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6044.132306] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6044.134823] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6044.138025] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6044.140218] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6044.142165] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6044.143829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6044.146547] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6044.148999] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6044.150914] lmv_intent_open+0x338/0x1150 [lmv] [ 6044.152587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6044.154869] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6044.157010] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6044.158909] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6044.161165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6044.163323] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6044.165402] ? do_raw_spin_unlock+0x75/0x190 [ 6044.167836] ? _raw_spin_unlock+0x12/0x30 [ 6044.170120] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6044.172282] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6044.174573] ll_atomic_open+0x36a/0xd70 [lustre] [ 6044.175683] lookup_open+0x356/0xb40 [ 6044.176542] do_last+0x313/0xfc0 [ 6044.177396] ? nd_jump_root+0xe5/0x160 [ 6044.178841] ? path_init+0x437/0x520 [ 6044.180353] path_openat+0xf7/0x500 [ 6044.181914] ? do_raw_spin_unlock+0x75/0x190 [ 6044.183490] do_filp_open+0x99/0x140 [ 6044.185126] ? getname_flags+0x6e/0x330 [ 6044.186763] ? __check_object_size+0xff/0x256 [ 6044.188548] ? do_raw_spin_unlock+0x75/0x190 [ 6044.190279] ? _raw_spin_unlock+0x12/0x30 [ 6044.192098] do_sys_openat2+0x2b4/0x410 [ 6044.193763] do_sys_open+0x73/0xa0 [ 6044.195209] __x64_sys_openat+0x24/0x30 [ 6044.196818] do_syscall_64+0xc1/0x440 [ 6044.198236] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6044.200845] RIP: 0033:0x7fdc0afbf332 [ 6044.202441] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6044.209822] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6044.212480] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6044.215337] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6044.218503] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6044.221536] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6044.224366] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6052.055731] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6052.061477] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6052.063405] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6052.124847] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6052.124861] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6052.124868] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6052.125138] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6052.125148] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6052.125152] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6052.125156] RAX: 00000000000055de RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6052.125158] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6052.125160] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6052.125162] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6052.125164] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6052.125167] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6052.125169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6052.125171] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6052.125396] Call Trace: [ 6052.125402] [ 6052.125408] ? show_regs.cold.9+0x22/0x2f [ 6052.125417] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6052.125424] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6052.125429] ? watchdog+0x40/0x40 [ 6052.125438] ? hrtimer_interrupt+0x128/0x2c0 [ 6052.125443] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6052.125453] ? apic_timer_interrupt+0xf/0x20 [ 6052.125458] [ 6052.125475] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6052.125481] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6052.125555] do_raw_spin_lock+0x177/0x190 [ 6052.125560] _raw_spin_lock+0x21/0x30 [ 6052.125565] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6052.125786] process_one_work+0x2c8/0x700 [ 6052.125794] worker_thread+0x296/0x6e0 [ 6052.125798] ? rescuer_thread+0x570/0x570 [ 6052.125807] kthread+0x1d1/0x200 [ 6052.125812] ? set_kthread_struct+0x70/0x70 [ 6052.125815] ret_from_fork+0x1f/0x30 [ 6052.283844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6052.309487] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6052.317123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6052.326337] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6052.329676] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6052.353342] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6052.364243] RAX: 0000000000007d8a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6052.373449] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6052.379009] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6052.385431] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6052.392276] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6052.398754] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6052.402267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6052.404566] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6052.407402] Call Trace: [ 6052.408269] [ 6052.410235] ? show_regs.cold.9+0x22/0x2f [ 6052.412673] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6052.419433] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6052.424219] ? watchdog+0x40/0x40 [ 6052.429465] ? hrtimer_interrupt+0x128/0x2c0 [ 6052.430596] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6052.432332] ? apic_timer_interrupt+0xf/0x20 [ 6052.434196] [ 6052.434753] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6052.437224] do_raw_spin_lock+0x177/0x190 [ 6052.440228] _raw_spin_lock+0x21/0x30 [ 6052.442846] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6052.446147] ? do_raw_spin_unlock+0x75/0x190 [ 6052.449988] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6052.455745] ? do_wait_intr+0xf0/0xf0 [ 6052.458766] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6052.464004] mdc_close+0x53b/0xd70 [mdc] [ 6052.476024] lmv_close+0x18d/0x490 [lmv] [ 6052.484430] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6052.490091] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6052.496726] ll_md_close+0x274/0x800 [lustre] [ 6052.505415] ll_file_release+0x128/0xc50 [lustre] [ 6052.516777] __fput+0x10d/0x440 [ 6052.519445] ____fput+0x12/0x20 [ 6052.526398] task_work_run+0xc6/0x110 [ 6052.530349] exit_to_usermode_loop+0x1dd/0x1f0 [ 6052.531792] do_syscall_64+0x3d6/0x440 [ 6052.536271] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6052.541679] RIP: 0033:0x7fdc0afbfcd8 [ 6052.548670] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6052.571821] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6052.582225] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6052.587968] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6052.597830] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6052.602107] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6052.615340] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6056.422734] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4031s! [ 6056.435335] Showing busy workqueues and worker pools: [ 6056.439023] workqueue events: flags=0x0 [ 6056.448983] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6056.458931] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6056.464309] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6056.468774] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6056.475044] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6056.484909] pending: kfree_rcu_monitor [ 6056.487036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6056.492680] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6056.502416] workqueue events_long: flags=0x0 [ 6056.504403] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6056.509043] pending: lu_objects_flush [obdclass] [ 6056.513099] workqueue events_unbound: flags=0x2 [ 6056.516031] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6056.520349] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6056.538783] workqueue events_power_efficient: flags=0x80 [ 6056.542137] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6056.546975] pending: neigh_periodic_work, check_lifetime [ 6056.549954] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6056.554324] pending: do_cache_clean, neigh_periodic_work [ 6056.558381] workqueue mm_percpu_wq: flags=0x8 [ 6056.562868] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6056.569353] pending: lru_add_drain_per_cpu [ 6056.574029] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6056.578672] pending: vmstat_update, lru_add_drain_per_cpu [ 6056.583904] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6056.593055] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6056.603854] workqueue writeback: flags=0x4a [ 6056.609370] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6056.615956] pending: wb_workfn, wb_workfn [ 6056.624307] workqueue xprtiod: flags=0xa [ 6056.630017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6056.638527] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6056.646796] workqueue ipv6_addrconf: flags=0x40008 [ 6056.653350] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6056.660090] pending: addrconf_verify_work [ 6056.666264] workqueue knbd0-recv: flags=0x1a [ 6056.680973] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6056.694137] in-flight: 655:recv_work [ 6056.697926] workqueue ptlrpc_pinger: flags=0x2 [ 6056.699339] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6056.708154] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6056.717081] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4031s workers=4 idle: 36396 38 11188 [ 6056.725060] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6047s workers=2 idle: 658 [ 6072.052052] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 6072.054217] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6072.077990] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6072.089921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6072.098758] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6072.102790] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6072.116974] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6072.121233] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6072.128820] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6072.132087] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6072.140084] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6072.147126] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6072.150736] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6072.156292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6072.160690] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6072.164615] Call Trace: [ 6072.166551] [ 6072.168577] ? show_regs.cold.9+0x22/0x2f [ 6072.171748] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6072.174862] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6072.180229] ? watchdog+0x40/0x40 [ 6072.182350] ? hrtimer_interrupt+0x128/0x2c0 [ 6072.189252] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6072.194856] ? apic_timer_interrupt+0xf/0x20 [ 6072.199706] [ 6072.203476] ? native_safe_halt+0x2e/0x50 [ 6072.206668] kvm_wait+0xcd/0xe0 [ 6072.209460] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6072.214443] do_raw_spin_lock+0x177/0x190 [ 6072.218072] _raw_spin_lock+0x21/0x30 [ 6072.220574] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6072.224435] ? do_raw_spin_unlock+0x75/0x190 [ 6072.231203] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6072.238799] ? do_wait_intr+0xf0/0xf0 [ 6072.240784] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6072.247816] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6072.252516] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6072.255696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6072.259515] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6072.267271] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6072.270769] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6072.274042] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6072.280930] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6072.284026] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6072.287444] lmv_intent_open+0x338/0x1150 [lmv] [ 6072.290554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6072.294024] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6072.297284] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6072.301302] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6072.307604] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6072.312604] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6072.320460] ? do_raw_spin_unlock+0x75/0x190 [ 6072.325296] ? _raw_spin_unlock+0x12/0x30 [ 6072.328152] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6072.331777] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6072.335753] ll_atomic_open+0x36a/0xd70 [lustre] [ 6072.339664] lookup_open+0x356/0xb40 [ 6072.345971] do_last+0x313/0xfc0 [ 6072.350122] ? nd_jump_root+0xe5/0x160 [ 6072.353708] ? path_init+0x437/0x520 [ 6072.359466] path_openat+0xf7/0x500 [ 6072.361751] ? do_raw_spin_unlock+0x75/0x190 [ 6072.364957] do_filp_open+0x99/0x140 [ 6072.368799] ? getname_flags+0x6e/0x330 [ 6072.369914] ? __check_object_size+0xff/0x256 [ 6072.376105] ? do_raw_spin_unlock+0x75/0x190 [ 6072.379638] ? _raw_spin_unlock+0x12/0x30 [ 6072.384120] do_sys_openat2+0x2b4/0x410 [ 6072.387520] do_sys_open+0x73/0xa0 [ 6072.390516] __x64_sys_openat+0x24/0x30 [ 6072.393937] do_syscall_64+0xc1/0x440 [ 6072.396424] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6072.402808] RIP: 0033:0x7fdc0afbf332 [ 6072.409700] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6072.431651] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6072.434207] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6072.455623] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6072.461436] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6072.472526] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6072.480821] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6080.051962] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6080.054075] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6080.064867] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6080.064946] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6080.064954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6080.064960] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6080.065226] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6080.065237] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6080.065241] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6080.065245] RAX: 0000000000007d70 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6080.065247] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6080.065249] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6080.065252] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6080.065254] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6080.065257] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6080.065260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6080.065263] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6080.065271] Call Trace: [ 6080.065277] [ 6080.065282] ? show_regs.cold.9+0x22/0x2f [ 6080.065292] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6080.065298] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6080.065302] ? watchdog+0x40/0x40 [ 6080.065306] ? hrtimer_interrupt+0x128/0x2c0 [ 6080.065310] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6080.065318] ? apic_timer_interrupt+0xf/0x20 [ 6080.065323] [ 6080.065326] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6080.065330] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6080.065336] do_raw_spin_lock+0x177/0x190 [ 6080.065341] _raw_spin_lock+0x21/0x30 [ 6080.065345] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6080.125075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6080.125150] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6080.125158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6080.125161] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6080.125174] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6080.125179] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6080.125183] RAX: 0000000000006784 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6080.125186] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6080.125188] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6080.125191] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6080.125193] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6080.125196] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6080.125199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6080.125202] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6080.125210] Call Trace: [ 6080.125216] [ 6080.125222] ? show_regs.cold.9+0x22/0x2f [ 6080.125232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6080.125238] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6080.125242] ? watchdog+0x40/0x40 [ 6080.125246] ? hrtimer_interrupt+0x128/0x2c0 [ 6080.125250] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6080.125258] ? apic_timer_interrupt+0xf/0x20 [ 6080.125263] [ 6080.125266] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6080.125273] do_raw_spin_lock+0x177/0x190 [ 6080.125278] _raw_spin_lock+0x21/0x30 [ 6080.125283] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6080.125568] ? do_raw_spin_unlock+0x75/0x190 [ 6080.125574] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6080.125767] ? do_wait_intr+0xf0/0xf0 [ 6080.125774] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6080.125968] mdc_close+0x53b/0xd70 [mdc] [ 6080.126003] lmv_close+0x18d/0x490 [lmv] [ 6080.126031] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6080.126147] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6080.126232] ll_md_close+0x274/0x800 [lustre] [ 6080.126314] ll_file_release+0x128/0xc50 [lustre] [ 6080.130182] __fput+0x10d/0x440 [ 6080.130195] ____fput+0x12/0x20 [ 6080.130200] task_work_run+0xc6/0x110 [ 6080.130208] exit_to_usermode_loop+0x1dd/0x1f0 [ 6080.130215] do_syscall_64+0x3d6/0x440 [ 6080.130220] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6080.130227] RIP: 0033:0x7fdc0afbfcd8 [ 6080.130234] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6080.130238] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6080.130243] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6080.130245] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6080.130247] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6080.130249] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6080.130251] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6080.681326] process_one_work+0x2c8/0x700 [ 6080.682656] worker_thread+0x296/0x6e0 [ 6080.687644] ? rescuer_thread+0x570/0x570 [ 6080.691667] kthread+0x1d1/0x200 [ 6080.695715] ? set_kthread_struct+0x70/0x70 [ 6080.698892] ret_from_fork+0x1f/0x30 [ 6084.048028] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 6084.051909] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6084.091762] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6084.108246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6084.114963] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6084.121584] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6084.133892] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6084.140315] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6084.144030] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6084.151102] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6084.154534] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6084.166365] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6084.179307] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6084.189203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6084.205065] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6084.217396] Call Trace: [ 6084.219754] [ 6084.226891] ? show_regs.cold.9+0x22/0x2f [ 6084.228429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6084.244726] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6084.253544] ? watchdog+0x40/0x40 [ 6084.256006] ? hrtimer_interrupt+0x128/0x2c0 [ 6084.261299] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6084.263135] ? apic_timer_interrupt+0xf/0x20 [ 6084.272396] [ 6084.275931] ? native_safe_halt+0x2e/0x50 [ 6084.285734] kvm_wait+0xcd/0xe0 [ 6084.287557] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6084.290610] ? dump_stack+0xf6/0x10e [ 6084.295562] do_raw_spin_lock+0x177/0x190 [ 6084.298614] _raw_spin_lock+0x21/0x30 [ 6084.304377] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6084.311463] obd_mod_free+0x46/0x140 [ptlrpc] [ 6084.320198] mdc_commit_open+0x91/0xb0 [mdc] [ 6084.326355] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6084.335843] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6084.343877] after_reply+0xe2c/0x1780 [ptlrpc] [ 6084.348904] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6084.352692] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6084.363756] ? do_wait_intr+0xf0/0xf0 [ 6084.366573] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6084.370750] kthread+0x1d1/0x200 [ 6084.373440] ? set_kthread_struct+0x70/0x70 [ 6084.375905] ret_from_fork+0x1f/0x30 [ 6087.136338] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4062s! [ 6087.143310] Showing busy workqueues and worker pools: [ 6087.148841] workqueue events: flags=0x0 [ 6087.151741] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6087.160189] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6087.162932] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6087.169841] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6087.180341] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6087.184928] pending: kfree_rcu_monitor [ 6087.187128] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6087.191839] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6087.213753] workqueue events_long: flags=0x0 [ 6087.216792] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6087.221181] pending: lu_objects_flush [obdclass] [ 6087.226700] workqueue events_unbound: flags=0x2 [ 6087.229791] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6087.231797] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6087.255535] workqueue events_power_efficient: flags=0x80 [ 6087.260851] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6087.271729] pending: neigh_periodic_work, check_lifetime [ 6087.273952] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6087.281998] pending: do_cache_clean, neigh_periodic_work [ 6087.286295] workqueue mm_percpu_wq: flags=0x8 [ 6087.298355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6087.302726] pending: lru_add_drain_per_cpu [ 6087.305631] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6087.317293] pending: vmstat_update, lru_add_drain_per_cpu [ 6087.329834] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6087.343995] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6087.353325] workqueue writeback: flags=0x4a [ 6087.363061] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6087.367429] pending: wb_workfn, wb_workfn [ 6087.369013] workqueue xprtiod: flags=0xa [ 6087.380563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6087.388929] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6087.401125] workqueue ipv6_addrconf: flags=0x40008 [ 6087.404697] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6087.411726] pending: addrconf_verify_work [ 6087.415371] workqueue knbd0-recv: flags=0x1a [ 6087.416977] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6087.427536] in-flight: 655:recv_work [ 6087.434720] workqueue ptlrpc_pinger: flags=0x2 [ 6087.436418] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6087.446648] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6087.452141] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4062s workers=4 idle: 36396 38 11188 [ 6087.460676] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6078s workers=2 idle: 658 [ 6100.052253] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 6100.055164] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6100.082232] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6100.093332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6100.097349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6100.099364] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6100.113007] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6100.115556] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6100.118228] RDX: 00000000000069e7 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6100.120828] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6100.123516] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6100.126398] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6100.128894] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6100.132025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6100.134219] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6100.138202] Call Trace: [ 6100.139742] [ 6100.140736] ? show_regs.cold.9+0x22/0x2f [ 6100.142014] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6100.143515] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6100.153431] ? watchdog+0x40/0x40 [ 6100.154974] ? hrtimer_interrupt+0x128/0x2c0 [ 6100.162375] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6100.171527] ? apic_timer_interrupt+0xf/0x20 [ 6100.175897] [ 6100.176611] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6100.183358] do_raw_spin_lock+0x177/0x190 [ 6100.185289] _raw_spin_lock+0x21/0x30 [ 6100.187522] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6100.191241] ? do_raw_spin_unlock+0x75/0x190 [ 6100.196457] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6100.202592] ? do_wait_intr+0xf0/0xf0 [ 6100.204270] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6100.209232] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6100.212546] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6100.223077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6100.228903] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6100.231599] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6100.240994] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6100.244529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6100.247038] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6100.250209] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6100.253400] lmv_intent_open+0x338/0x1150 [lmv] [ 6100.255408] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6100.258242] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6100.260795] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6100.262344] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6100.265295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6100.268242] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6100.269886] ? do_raw_spin_unlock+0x75/0x190 [ 6100.271478] ? _raw_spin_unlock+0x12/0x30 [ 6100.273120] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6100.275115] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6100.277122] ll_atomic_open+0x36a/0xd70 [lustre] [ 6100.280162] lookup_open+0x356/0xb40 [ 6100.281422] do_last+0x313/0xfc0 [ 6100.287478] ? nd_jump_root+0xe5/0x160 [ 6100.293800] ? path_init+0x437/0x520 [ 6100.295580] path_openat+0xf7/0x500 [ 6100.296778] ? do_raw_spin_unlock+0x75/0x190 [ 6100.303082] do_filp_open+0x99/0x140 [ 6100.305017] ? getname_flags+0x6e/0x330 [ 6100.311496] ? __check_object_size+0xff/0x256 [ 6100.313617] ? do_raw_spin_unlock+0x75/0x190 [ 6100.315319] ? _raw_spin_unlock+0x12/0x30 [ 6100.319476] do_sys_openat2+0x2b4/0x410 [ 6100.321675] do_sys_open+0x73/0xa0 [ 6100.323386] __x64_sys_openat+0x24/0x30 [ 6100.325286] do_syscall_64+0xc1/0x440 [ 6100.328250] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6100.330508] RIP: 0033:0x7fdc0afbf332 [ 6100.336136] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6100.351346] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6100.358574] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6100.366287] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6100.370372] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6100.373977] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6100.377399] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6108.051074] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6108.054798] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6108.056191] Modules linked in: [ 6108.062021] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6108.062098] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6108.062105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6108.062108] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6108.062122] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6108.062127] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6108.062131] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6108.062133] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6108.062135] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6108.062137] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6108.062139] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6108.062142] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6108.062146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6108.062148] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6108.062157] Call Trace: [ 6108.062162] [ 6108.062168] ? show_regs.cold.9+0x22/0x2f [ 6108.062177] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6108.062184] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6108.062189] ? watchdog+0x40/0x40 [ 6108.062194] ? hrtimer_interrupt+0x128/0x2c0 [ 6108.062198] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6108.062205] ? apic_timer_interrupt+0xf/0x20 [ 6108.062210] [ 6108.062213] ? native_safe_halt+0x2e/0x50 [ 6108.062217] kvm_wait+0xcd/0xe0 [ 6108.062224] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6108.062233] do_raw_spin_lock+0x177/0x190 [ 6108.062238] _raw_spin_lock+0x21/0x30 [ 6108.062242] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6108.062515] ? do_raw_spin_unlock+0x75/0x190 [ 6108.062521] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6108.062716] ? do_wait_intr+0xf0/0xf0 [ 6108.062725] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6108.073839] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6108.073928] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6108.073936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6108.073942] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6108.074231] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6108.074243] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6108.074247] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6108.074252] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6108.074254] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6108.074257] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6108.074259] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6108.074262] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6108.074266] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6108.074269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6108.074272] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6108.074281] Call Trace: [ 6108.074287] [ 6108.074292] ? show_regs.cold.9+0x22/0x2f [ 6108.074302] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6108.074309] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6108.074313] ? watchdog+0x40/0x40 [ 6108.074317] ? hrtimer_interrupt+0x128/0x2c0 [ 6108.074320] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6108.074327] ? apic_timer_interrupt+0xf/0x20 [ 6108.074331] [ 6108.074334] ? native_safe_halt+0x2e/0x50 [ 6108.074338] kvm_wait+0xcd/0xe0 [ 6108.074346] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6108.074354] do_raw_spin_lock+0x177/0x190 [ 6108.074358] _raw_spin_lock+0x21/0x30 [ 6108.074362] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6108.142428] mdc_close+0x53b/0xd70 [mdc] [ 6108.154358] process_one_work+0x2c8/0x700 [ 6108.154378] worker_thread+0x296/0x6e0 [ 6108.154383] ? rescuer_thread+0x570/0x570 [ 6108.154387] kthread+0x1d1/0x200 [ 6108.154392] ? set_kthread_struct+0x70/0x70 [ 6108.154395] ret_from_fork+0x1f/0x30 [ 6108.522579] lmv_close+0x18d/0x490 [lmv] [ 6108.525086] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6108.527137] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6108.529831] ll_md_close+0x274/0x800 [lustre] [ 6108.531263] ll_file_release+0x128/0xc50 [lustre] [ 6108.533072] __fput+0x10d/0x440 [ 6108.534711] ____fput+0x12/0x20 [ 6108.536299] task_work_run+0xc6/0x110 [ 6108.537701] exit_to_usermode_loop+0x1dd/0x1f0 [ 6108.539196] do_syscall_64+0x3d6/0x440 [ 6108.541090] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6108.542710] RIP: 0033:0x7fdc0afbfcd8 [ 6108.545180] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6108.555685] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6108.559389] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6108.564230] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6108.568697] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6108.574144] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6108.577393] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6112.048554] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 6112.057889] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6112.085037] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6112.092546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6112.098108] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6112.101971] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6112.111074] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6112.114605] RAX: 0000000000007f83 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6112.118217] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6112.122018] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6112.124988] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6112.128219] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6112.130208] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6112.133392] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6112.135406] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6112.139364] Call Trace: [ 6112.140562] [ 6112.141307] ? show_regs.cold.9+0x22/0x2f [ 6112.142852] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6112.144492] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6112.145767] ? watchdog+0x40/0x40 [ 6112.147699] ? hrtimer_interrupt+0x128/0x2c0 [ 6112.150847] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6112.152781] ? apic_timer_interrupt+0xf/0x20 [ 6112.154684] [ 6112.155609] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6112.157921] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6112.160352] ? dump_stack+0xf6/0x10e [ 6112.161512] do_raw_spin_lock+0x177/0x190 [ 6112.163874] _raw_spin_lock+0x21/0x30 [ 6112.165344] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6112.167976] obd_mod_free+0x46/0x140 [ptlrpc] [ 6112.170204] mdc_commit_open+0x91/0xb0 [mdc] [ 6112.172061] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6112.174444] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6112.176487] after_reply+0xe2c/0x1780 [ptlrpc] [ 6112.179648] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6112.182916] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6112.185793] ? do_wait_intr+0xf0/0xf0 [ 6112.187491] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6112.190647] kthread+0x1d1/0x200 [ 6112.193227] ? set_kthread_struct+0x70/0x70 [ 6112.194661] ret_from_fork+0x1f/0x30 [ 6117.855092] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4092s! [ 6117.858802] Showing busy workqueues and worker pools: [ 6117.860472] workqueue events: flags=0x0 [ 6117.862023] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6117.864834] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6117.867950] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6117.871074] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6117.875885] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6117.878943] pending: kfree_rcu_monitor [ 6117.880581] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6117.883433] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6117.888970] workqueue events_long: flags=0x0 [ 6117.890927] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6117.893972] pending: lu_objects_flush [obdclass] [ 6117.896606] workqueue events_unbound: flags=0x2 [ 6117.898121] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6117.900225] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6117.904556] workqueue events_power_efficient: flags=0x80 [ 6117.906985] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6117.909640] pending: neigh_periodic_work, check_lifetime [ 6117.912396] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6117.915695] pending: do_cache_clean, neigh_periodic_work [ 6117.918292] workqueue mm_percpu_wq: flags=0x8 [ 6117.919970] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6117.922466] pending: lru_add_drain_per_cpu [ 6117.924568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6117.927792] pending: vmstat_update, lru_add_drain_per_cpu [ 6117.929498] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6117.931556] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6117.933421] workqueue writeback: flags=0x4a [ 6117.935715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6117.938916] pending: wb_workfn, wb_workfn [ 6117.941443] workqueue xprtiod: flags=0xa [ 6117.945809] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6117.950895] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6117.957884] workqueue ipv6_addrconf: flags=0x40008 [ 6117.959859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6117.963676] pending: addrconf_verify_work [ 6117.966333] workqueue knbd0-recv: flags=0x1a [ 6117.972085] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6117.977804] in-flight: 655:recv_work [ 6117.980062] workqueue ptlrpc_pinger: flags=0x2 [ 6117.982105] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6117.988317] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6117.994768] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4093s workers=4 idle: 36396 38 11188 [ 6118.001223] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6108s workers=2 idle: 658 [ 6128.052690] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 6128.058949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6128.116320] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6128.128234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6128.136439] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6128.139859] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6128.170730] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6128.180826] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6128.189198] RDX: 0000000000002b81 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6128.193253] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6128.198523] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6128.208747] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6128.219888] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6128.225215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6128.236120] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6128.249737] Call Trace: [ 6128.254607] [ 6128.255667] ? show_regs.cold.9+0x22/0x2f [ 6128.261958] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6128.276832] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6128.284499] ? watchdog+0x40/0x40 [ 6128.291719] ? hrtimer_interrupt+0x128/0x2c0 [ 6128.297038] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6128.306519] ? apic_timer_interrupt+0xf/0x20 [ 6128.309741] [ 6128.314870] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6128.319895] do_raw_spin_lock+0x177/0x190 [ 6128.326601] _raw_spin_lock+0x21/0x30 [ 6128.330476] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6128.336470] ? do_raw_spin_unlock+0x75/0x190 [ 6128.346677] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6128.348775] ? do_wait_intr+0xf0/0xf0 [ 6128.361672] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6128.372892] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6128.383691] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6128.395740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6128.403533] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6128.421479] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6128.436293] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6128.438468] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6128.448653] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6128.457317] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6128.469725] lmv_intent_open+0x338/0x1150 [lmv] [ 6128.479513] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6128.490759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6128.493925] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6128.497201] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6128.501770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6128.510234] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6128.514475] ? do_raw_spin_unlock+0x75/0x190 [ 6128.520187] ? _raw_spin_unlock+0x12/0x30 [ 6128.521703] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6128.535398] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6128.541416] ll_atomic_open+0x36a/0xd70 [lustre] [ 6128.545032] lookup_open+0x356/0xb40 [ 6128.546456] do_last+0x313/0xfc0 [ 6128.548722] ? nd_jump_root+0xe5/0x160 [ 6128.550531] ? path_init+0x437/0x520 [ 6128.552815] path_openat+0xf7/0x500 [ 6128.554031] ? do_raw_spin_unlock+0x75/0x190 [ 6128.555523] do_filp_open+0x99/0x140 [ 6128.568747] ? getname_flags+0x6e/0x330 [ 6128.575637] ? __check_object_size+0xff/0x256 [ 6128.582567] ? do_raw_spin_unlock+0x75/0x190 [ 6128.589689] ? _raw_spin_unlock+0x12/0x30 [ 6128.592062] do_sys_openat2+0x2b4/0x410 [ 6128.601259] do_sys_open+0x73/0xa0 [ 6128.602087] __x64_sys_openat+0x24/0x30 [ 6128.603171] do_syscall_64+0xc1/0x440 [ 6128.604253] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6128.617152] RIP: 0033:0x7fdc0afbf332 [ 6128.618296] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6128.651325] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6128.665187] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6128.671781] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6128.680789] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6128.694708] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6128.705539] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6136.051031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6136.054727] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6136.061550] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6136.078044] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6136.078060] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6136.078068] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6136.078474] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6136.078486] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6136.078490] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6136.078494] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6136.078496] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6136.078499] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6136.078501] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6136.078503] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6136.078507] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6136.078510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6136.078517] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6136.078526] Call Trace: [ 6136.078532] [ 6136.078537] ? show_regs.cold.9+0x22/0x2f [ 6136.078547] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6136.078553] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6136.078558] ? watchdog+0x40/0x40 [ 6136.078561] ? hrtimer_interrupt+0x128/0x2c0 [ 6136.078571] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6136.078578] ? apic_timer_interrupt+0xf/0x20 [ 6136.078583] [ 6136.078586] ? native_safe_halt+0x2e/0x50 [ 6136.078590] kvm_wait+0xcd/0xe0 [ 6136.078598] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6136.078605] do_raw_spin_lock+0x177/0x190 [ 6136.078609] _raw_spin_lock+0x21/0x30 [ 6136.078613] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6136.079093] process_one_work+0x2c8/0x700 [ 6136.079101] worker_thread+0x296/0x6e0 [ 6136.079105] ? rescuer_thread+0x570/0x570 [ 6136.079109] kthread+0x1d1/0x200 [ 6136.079113] ? set_kthread_struct+0x70/0x70 [ 6136.079117] ret_from_fork+0x1f/0x30 [ 6136.328484] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6136.380701] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6136.388920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6136.398907] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6136.405925] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6136.421091] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6136.433753] RAX: 0000000000002484 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6136.441943] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6136.448956] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6136.454514] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6136.460919] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6136.465744] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6136.473120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6136.477683] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6136.482938] Call Trace: [ 6136.485406] [ 6136.486629] ? show_regs.cold.9+0x22/0x2f [ 6136.489408] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6136.492859] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6136.495196] ? watchdog+0x40/0x40 [ 6136.503009] ? hrtimer_interrupt+0x128/0x2c0 [ 6136.506876] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6136.516332] ? apic_timer_interrupt+0xf/0x20 [ 6136.521424] [ 6136.525657] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6136.533000] do_raw_spin_lock+0x177/0x190 [ 6136.542427] _raw_spin_lock+0x21/0x30 [ 6136.544393] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6136.548057] ? do_raw_spin_unlock+0x75/0x190 [ 6136.550836] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6136.553671] ? do_wait_intr+0xf0/0xf0 [ 6136.555383] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6136.566281] mdc_close+0x53b/0xd70 [mdc] [ 6136.575829] lmv_close+0x18d/0x490 [lmv] [ 6136.579083] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6136.587225] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6136.599168] ll_md_close+0x274/0x800 [lustre] [ 6136.603932] ll_file_release+0x128/0xc50 [lustre] [ 6136.606567] __fput+0x10d/0x440 [ 6136.609996] ____fput+0x12/0x20 [ 6136.611709] task_work_run+0xc6/0x110 [ 6136.619495] exit_to_usermode_loop+0x1dd/0x1f0 [ 6136.625246] do_syscall_64+0x3d6/0x440 [ 6136.628719] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6136.631430] RIP: 0033:0x7fdc0afbfcd8 [ 6136.634020] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6136.648976] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6136.658119] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6136.668222] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6136.683912] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6136.697363] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6136.701511] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6144.048553] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 6144.053332] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6144.084877] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6144.092732] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6144.097566] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6144.100216] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6144.109867] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6144.117486] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6144.120615] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6144.123166] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6144.128110] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6144.131042] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6144.134423] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6144.139302] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6144.142165] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6144.149515] Call Trace: [ 6144.152254] [ 6144.153771] ? show_regs.cold.9+0x22/0x2f [ 6144.155459] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6144.160129] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6144.164219] ? watchdog+0x40/0x40 [ 6144.165829] ? hrtimer_interrupt+0x128/0x2c0 [ 6144.170102] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6144.172077] ? apic_timer_interrupt+0xf/0x20 [ 6144.174169] [ 6144.176272] ? native_safe_halt+0x2e/0x50 [ 6144.178583] kvm_wait+0xcd/0xe0 [ 6144.181894] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6144.185287] ? dump_stack+0xf6/0x10e [ 6144.187251] do_raw_spin_lock+0x177/0x190 [ 6144.189126] _raw_spin_lock+0x21/0x30 [ 6144.190488] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6144.193227] obd_mod_free+0x46/0x140 [ptlrpc] [ 6144.197146] mdc_commit_open+0x91/0xb0 [mdc] [ 6144.201247] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6144.204005] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6144.208239] after_reply+0xe2c/0x1780 [ptlrpc] [ 6144.210578] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6144.213415] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6144.216985] ? do_wait_intr+0xf0/0xf0 [ 6144.218746] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6144.221150] kthread+0x1d1/0x200 [ 6144.222583] ? set_kthread_struct+0x70/0x70 [ 6144.225973] ret_from_fork+0x1f/0x30 [ 6148.063705] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4123s! [ 6148.069670] Showing busy workqueues and worker pools: [ 6148.072464] workqueue events: flags=0x0 [ 6148.075899] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6148.079687] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6148.083811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6148.087374] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6148.094333] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6148.101018] pending: kfree_rcu_monitor [ 6148.102520] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6148.104885] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6148.113889] workqueue events_long: flags=0x0 [ 6148.116881] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6148.121720] pending: lu_objects_flush [obdclass] [ 6148.123759] workqueue events_unbound: flags=0x2 [ 6148.126208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6148.130911] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6148.136240] workqueue events_power_efficient: flags=0x80 [ 6148.139012] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6148.145967] pending: neigh_periodic_work, check_lifetime [ 6148.147727] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6148.151350] pending: do_cache_clean, neigh_periodic_work [ 6148.153550] workqueue mm_percpu_wq: flags=0x8 [ 6148.155941] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6148.159015] pending: lru_add_drain_per_cpu [ 6148.161033] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6148.163794] pending: vmstat_update, lru_add_drain_per_cpu [ 6148.166171] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6148.169970] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6148.172558] workqueue writeback: flags=0x4a [ 6148.175347] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6148.179472] pending: wb_workfn, wb_workfn [ 6148.181125] workqueue xprtiod: flags=0xa [ 6148.182947] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6148.188695] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6148.193737] workqueue ipv6_addrconf: flags=0x40008 [ 6148.196746] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6148.200794] pending: addrconf_verify_work [ 6148.203456] workqueue knbd0-recv: flags=0x1a [ 6148.205350] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6148.208196] in-flight: 655:recv_work [ 6148.211360] workqueue ptlrpc_pinger: flags=0x2 [ 6148.213566] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6148.218461] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6148.221380] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4123s workers=4 idle: 36396 38 11188 [ 6148.225990] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6138s workers=2 idle: 658 [ 6156.055218] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 6156.059366] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6156.086833] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6156.090888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6156.093257] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6156.096008] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6156.105172] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6156.108903] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6156.113265] RDX: 000000000000038e RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6156.116874] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6156.121371] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6156.124043] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6156.126727] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6156.129143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6156.131402] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6156.134024] Call Trace: [ 6156.135194] [ 6156.135883] ? show_regs.cold.9+0x22/0x2f [ 6156.137328] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6156.139092] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6156.140620] ? watchdog+0x40/0x40 [ 6156.142008] ? hrtimer_interrupt+0x128/0x2c0 [ 6156.143429] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6156.145038] ? apic_timer_interrupt+0xf/0x20 [ 6156.146602] [ 6156.147438] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6156.149379] do_raw_spin_lock+0x177/0x190 [ 6156.150923] _raw_spin_lock+0x21/0x30 [ 6156.152081] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6156.154419] ? do_raw_spin_unlock+0x75/0x190 [ 6156.156614] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6156.158806] ? do_wait_intr+0xf0/0xf0 [ 6156.160447] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6156.162935] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6156.165980] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6156.169307] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6156.171992] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6156.174535] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6156.175952] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6156.178129] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6156.181552] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6156.183948] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6156.186194] lmv_intent_open+0x338/0x1150 [lmv] [ 6156.187970] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6156.190384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6156.192504] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6156.194220] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6156.196108] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6156.198533] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6156.199962] ? do_raw_spin_unlock+0x75/0x190 [ 6156.201131] ? _raw_spin_unlock+0x12/0x30 [ 6156.202837] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6156.204607] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6156.207315] ll_atomic_open+0x36a/0xd70 [lustre] [ 6156.210173] lookup_open+0x356/0xb40 [ 6156.211605] do_last+0x313/0xfc0 [ 6156.213175] ? nd_jump_root+0xe5/0x160 [ 6156.214434] ? path_init+0x437/0x520 [ 6156.216020] path_openat+0xf7/0x500 [ 6156.218061] ? do_raw_spin_unlock+0x75/0x190 [ 6156.219533] do_filp_open+0x99/0x140 [ 6156.221830] ? getname_flags+0x6e/0x330 [ 6156.223258] ? __check_object_size+0xff/0x256 [ 6156.225088] ? do_raw_spin_unlock+0x75/0x190 [ 6156.227166] ? _raw_spin_unlock+0x12/0x30 [ 6156.229296] do_sys_openat2+0x2b4/0x410 [ 6156.231087] do_sys_open+0x73/0xa0 [ 6156.233648] __x64_sys_openat+0x24/0x30 [ 6156.236183] do_syscall_64+0xc1/0x440 [ 6156.237909] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6156.239442] RIP: 0033:0x7fdc0afbf332 [ 6156.242831] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6156.253471] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6156.259931] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6156.262794] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6156.270340] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6156.276487] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6156.279669] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6164.051108] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6164.056576] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6164.057390] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6164.081151] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6164.081168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6164.081176] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6164.081541] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6164.081559] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6164.081563] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6164.081568] RAX: 0000000000003c14 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6164.081570] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6164.081573] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6164.081575] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6164.081578] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6164.081581] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6164.081584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6164.081586] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6164.081594] Call Trace: [ 6164.081600] [ 6164.081605] ? show_regs.cold.9+0x22/0x2f [ 6164.081615] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6164.081622] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6164.081626] ? watchdog+0x40/0x40 [ 6164.081630] ? hrtimer_interrupt+0x128/0x2c0 [ 6164.081634] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6164.081642] ? apic_timer_interrupt+0xf/0x20 [ 6164.081647] [ 6164.081650] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6164.081655] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6164.081661] do_raw_spin_lock+0x177/0x190 [ 6164.081667] _raw_spin_lock+0x21/0x30 [ 6164.081679] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6164.082141] process_one_work+0x2c8/0x700 [ 6164.082150] worker_thread+0x296/0x6e0 [ 6164.082156] ? rescuer_thread+0x570/0x570 [ 6164.082160] kthread+0x1d1/0x200 [ 6164.082165] ? set_kthread_struct+0x70/0x70 [ 6164.082168] ret_from_fork+0x1f/0x30 [ 6164.237826] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6164.264607] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6164.279294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6164.287778] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6164.290463] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6164.299733] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6164.303054] RAX: 000000000000260c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6164.305824] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6164.313591] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6164.324809] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6164.332228] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6164.342274] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6164.347118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6164.356451] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6164.361096] Call Trace: [ 6164.362129] [ 6164.363255] ? show_regs.cold.9+0x22/0x2f [ 6164.365782] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6164.368652] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6164.372392] ? watchdog+0x40/0x40 [ 6164.374832] ? hrtimer_interrupt+0x128/0x2c0 [ 6164.377126] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6164.379445] ? apic_timer_interrupt+0xf/0x20 [ 6164.386648] [ 6164.387627] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6164.391054] do_raw_spin_lock+0x177/0x190 [ 6164.394220] _raw_spin_lock+0x21/0x30 [ 6164.398673] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6164.412314] ? do_raw_spin_unlock+0x75/0x190 [ 6164.415218] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6164.419202] ? do_wait_intr+0xf0/0xf0 [ 6164.421995] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6164.424997] mdc_close+0x53b/0xd70 [mdc] [ 6164.427030] lmv_close+0x18d/0x490 [lmv] [ 6164.438878] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6164.442399] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6164.448092] ll_md_close+0x274/0x800 [lustre] [ 6164.454531] ll_file_release+0x128/0xc50 [lustre] [ 6164.456371] __fput+0x10d/0x440 [ 6164.462203] ____fput+0x12/0x20 [ 6164.465049] task_work_run+0xc6/0x110 [ 6164.468084] exit_to_usermode_loop+0x1dd/0x1f0 [ 6164.470732] do_syscall_64+0x3d6/0x440 [ 6164.473155] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6164.477227] RIP: 0033:0x7fdc0afbfcd8 [ 6164.480155] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6164.499802] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6164.503615] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6164.505482] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6164.509932] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6164.515144] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6164.518234] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6176.048023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6176.052506] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6176.077087] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6176.082905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6176.086569] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6176.088667] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6176.096919] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6176.100115] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6176.102953] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6176.106116] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6176.109419] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6176.114184] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6176.120182] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6176.127992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6176.136101] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6176.140203] Call Trace: [ 6176.141547] [ 6176.142627] ? show_regs.cold.9+0x22/0x2f [ 6176.144794] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6176.147548] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6176.150119] ? watchdog+0x40/0x40 [ 6176.152038] ? hrtimer_interrupt+0x128/0x2c0 [ 6176.154190] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6176.155932] ? apic_timer_interrupt+0xf/0x20 [ 6176.157706] [ 6176.158767] ? native_safe_halt+0x2e/0x50 [ 6176.160295] kvm_wait+0xcd/0xe0 [ 6176.161777] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6176.164802] ? dump_stack+0xf6/0x10e [ 6176.166680] do_raw_spin_lock+0x177/0x190 [ 6176.168318] _raw_spin_lock+0x21/0x30 [ 6176.170107] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6176.173013] obd_mod_free+0x46/0x140 [ptlrpc] [ 6176.175290] mdc_commit_open+0x91/0xb0 [mdc] [ 6176.177402] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6176.180384] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6176.183047] after_reply+0xe2c/0x1780 [ptlrpc] [ 6176.184571] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6176.186232] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6176.190338] ? do_wait_intr+0xf0/0xf0 [ 6176.191355] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6176.193013] kthread+0x1d1/0x200 [ 6176.194094] ? set_kthread_struct+0x70/0x70 [ 6176.195140] ret_from_fork+0x1f/0x30 [ 6178.284801] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4153s! [ 6178.290732] Showing busy workqueues and worker pools: [ 6178.294009] workqueue events: flags=0x0 [ 6178.298950] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6178.302304] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6178.305455] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6178.307640] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6178.312948] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6178.318418] pending: kfree_rcu_monitor [ 6178.320874] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6178.323566] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6178.328909] workqueue events_long: flags=0x0 [ 6178.332837] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6178.345973] pending: lu_objects_flush [obdclass] [ 6178.349221] workqueue events_unbound: flags=0x2 [ 6178.357737] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6178.362947] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6178.370359] workqueue events_power_efficient: flags=0x80 [ 6178.375961] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6178.381145] pending: neigh_periodic_work, check_lifetime [ 6178.387659] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6178.391451] pending: do_cache_clean, neigh_periodic_work [ 6178.395438] workqueue mm_percpu_wq: flags=0x8 [ 6178.397193] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6178.402197] pending: lru_add_drain_per_cpu [ 6178.404731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6178.411014] pending: vmstat_update, lru_add_drain_per_cpu [ 6178.414081] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6178.421051] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6178.429674] workqueue writeback: flags=0x4a [ 6178.430773] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6178.433142] pending: wb_workfn, wb_workfn [ 6178.434857] workqueue xprtiod: flags=0xa [ 6178.436606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6178.439232] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6178.445541] workqueue ipv6_addrconf: flags=0x40008 [ 6178.446644] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6178.449470] pending: addrconf_verify_work [ 6178.450882] workqueue knbd0-recv: flags=0x1a [ 6178.457622] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6178.461237] in-flight: 655:recv_work [ 6178.463995] workqueue ptlrpc_pinger: flags=0x2 [ 6178.467620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6178.472057] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6178.474795] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4153s workers=4 idle: 36396 38 11188 [ 6178.478191] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6169s workers=2 idle: 658 [ 6184.052299] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 6184.057119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6184.078748] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6184.084372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6184.087887] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6184.090864] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6184.099832] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6184.103843] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6184.108393] RDX: 00000000000056bd RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6184.111987] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6184.115367] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6184.118461] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6184.121433] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6184.125044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6184.127557] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6184.130965] Call Trace: [ 6184.132000] [ 6184.132986] ? show_regs.cold.9+0x22/0x2f [ 6184.134760] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6184.137171] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6184.139127] ? watchdog+0x40/0x40 [ 6184.141020] ? hrtimer_interrupt+0x128/0x2c0 [ 6184.143046] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6184.145442] ? apic_timer_interrupt+0xf/0x20 [ 6184.147398] [ 6184.148708] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6184.150989] do_raw_spin_lock+0x177/0x190 [ 6184.153068] _raw_spin_lock+0x21/0x30 [ 6184.154825] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6184.157360] ? do_raw_spin_unlock+0x75/0x190 [ 6184.159254] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6184.161752] ? do_wait_intr+0xf0/0xf0 [ 6184.163130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6184.165274] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6184.167672] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6184.170419] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6184.172835] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6184.175270] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6184.177313] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6184.179235] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6184.182149] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6184.184999] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6184.187736] lmv_intent_open+0x338/0x1150 [lmv] [ 6184.190077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6184.194352] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6184.197300] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6184.199381] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6184.201498] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6184.204115] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6184.206414] ? do_raw_spin_unlock+0x75/0x190 [ 6184.208563] ? _raw_spin_unlock+0x12/0x30 [ 6184.211062] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6184.213287] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6184.217514] ll_atomic_open+0x36a/0xd70 [lustre] [ 6184.220891] lookup_open+0x356/0xb40 [ 6184.222629] do_last+0x313/0xfc0 [ 6184.224005] ? nd_jump_root+0xe5/0x160 [ 6184.225816] ? path_init+0x437/0x520 [ 6184.227727] path_openat+0xf7/0x500 [ 6184.231830] ? do_raw_spin_unlock+0x75/0x190 [ 6184.234114] do_filp_open+0x99/0x140 [ 6184.236130] ? getname_flags+0x6e/0x330 [ 6184.239283] ? __check_object_size+0xff/0x256 [ 6184.242234] ? do_raw_spin_unlock+0x75/0x190 [ 6184.244946] ? _raw_spin_unlock+0x12/0x30 [ 6184.247569] do_sys_openat2+0x2b4/0x410 [ 6184.250237] do_sys_open+0x73/0xa0 [ 6184.252618] __x64_sys_openat+0x24/0x30 [ 6184.259018] do_syscall_64+0xc1/0x440 [ 6184.263732] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6184.268820] RIP: 0033:0x7fdc0afbf332 [ 6184.271904] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6184.282129] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6184.286694] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6184.290326] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6184.296236] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6184.300229] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6184.305682] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6192.051092] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6192.054243] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6192.054873] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6192.073450] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6192.073464] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6192.073470] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6192.077473] Modules linked in: [ 6192.083809] [ 6192.083818] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6192.083840] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6192.083845] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6192.083850] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6192.083852] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6192.083854] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6192.083856] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6192.083858] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6192.083861] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6192.083864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6192.083866] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6192.083875] Call Trace: [ 6192.083881] [ 6192.083886] ? show_regs.cold.9+0x22/0x2f [ 6192.083897] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6192.083903] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6192.083907] ? watchdog+0x40/0x40 [ 6192.083912] ? hrtimer_interrupt+0x128/0x2c0 [ 6192.083917] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6192.083923] ? apic_timer_interrupt+0xf/0x20 [ 6192.083928] [ 6192.083931] ? native_safe_halt+0x2e/0x50 [ 6192.083935] kvm_wait+0xcd/0xe0 [ 6192.083943] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6192.083955] do_raw_spin_lock+0x177/0x190 [ 6192.083961] _raw_spin_lock+0x21/0x30 [ 6192.083965] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6192.084277] process_one_work+0x2c8/0x700 [ 6192.084285] worker_thread+0x296/0x6e0 [ 6192.084290] ? rescuer_thread+0x570/0x570 [ 6192.084294] kthread+0x1d1/0x200 [ 6192.084298] ? set_kthread_struct+0x70/0x70 [ 6192.084301] ret_from_fork+0x1f/0x30 [ 6192.200574] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6192.222400] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6192.230999] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6192.235930] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6192.238537] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6192.249480] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6192.253577] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6192.257293] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6192.260374] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6192.263452] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6192.268665] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6192.272402] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6192.276157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6192.278581] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6192.282292] Call Trace: [ 6192.283177] [ 6192.283916] ? show_regs.cold.9+0x22/0x2f [ 6192.285847] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6192.287696] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6192.289765] ? watchdog+0x40/0x40 [ 6192.292225] ? hrtimer_interrupt+0x128/0x2c0 [ 6192.294711] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6192.298786] ? apic_timer_interrupt+0xf/0x20 [ 6192.302369] [ 6192.304146] ? native_safe_halt+0x2e/0x50 [ 6192.306401] kvm_wait+0xcd/0xe0 [ 6192.307821] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6192.310201] do_raw_spin_lock+0x177/0x190 [ 6192.312178] _raw_spin_lock+0x21/0x30 [ 6192.313866] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6192.316461] ? do_raw_spin_unlock+0x75/0x190 [ 6192.319993] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6192.322280] ? do_wait_intr+0xf0/0xf0 [ 6192.324118] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6192.325906] mdc_close+0x53b/0xd70 [mdc] [ 6192.328329] lmv_close+0x18d/0x490 [lmv] [ 6192.330541] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6192.335135] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6192.339908] ll_md_close+0x274/0x800 [lustre] [ 6192.342883] ll_file_release+0x128/0xc50 [lustre] [ 6192.345015] __fput+0x10d/0x440 [ 6192.346770] ____fput+0x12/0x20 [ 6192.349567] task_work_run+0xc6/0x110 [ 6192.351625] exit_to_usermode_loop+0x1dd/0x1f0 [ 6192.355553] do_syscall_64+0x3d6/0x440 [ 6192.358504] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6192.360240] RIP: 0033:0x7fdc0afbfcd8 [ 6192.361431] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6192.372540] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6192.377477] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6192.382928] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6192.385574] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6192.389251] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6192.392905] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6204.048197] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6204.052270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6204.074354] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6204.079582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6204.083395] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6204.085668] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6204.095653] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6204.098766] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6204.102164] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6204.106482] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6204.110698] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6204.114378] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6204.117970] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6204.121961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6204.125111] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6204.129219] Call Trace: [ 6204.130367] [ 6204.131282] ? show_regs.cold.9+0x22/0x2f [ 6204.133320] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6204.135289] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6204.137583] ? watchdog+0x40/0x40 [ 6204.138974] ? hrtimer_interrupt+0x128/0x2c0 [ 6204.141319] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6204.144609] ? apic_timer_interrupt+0xf/0x20 [ 6204.146431] [ 6204.147227] ? native_safe_halt+0x2e/0x50 [ 6204.148912] kvm_wait+0xcd/0xe0 [ 6204.150075] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6204.152458] ? dump_stack+0xf6/0x10e [ 6204.154100] do_raw_spin_lock+0x177/0x190 [ 6204.155511] _raw_spin_lock+0x21/0x30 [ 6204.156896] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6204.159076] obd_mod_free+0x46/0x140 [ptlrpc] [ 6204.161128] mdc_commit_open+0x91/0xb0 [mdc] [ 6204.162992] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6204.165384] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6204.167620] after_reply+0xe2c/0x1780 [ptlrpc] [ 6204.169721] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6204.172221] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6204.174181] ? do_wait_intr+0xf0/0xf0 [ 6204.175664] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6204.178271] kthread+0x1d1/0x200 [ 6204.179626] ? set_kthread_struct+0x70/0x70 [ 6204.181198] ret_from_fork+0x1f/0x30 [ 6208.992409] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4184s! [ 6208.998923] Showing busy workqueues and worker pools: [ 6209.003100] workqueue events: flags=0x0 [ 6209.007376] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6209.016303] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6209.019558] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6209.028369] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6209.044332] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6209.049468] pending: kfree_rcu_monitor [ 6209.054416] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6209.063091] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6209.077411] workqueue events_long: flags=0x0 [ 6209.082303] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6209.086298] pending: lu_objects_flush [obdclass] [ 6209.089183] workqueue events_unbound: flags=0x2 [ 6209.096380] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6209.099816] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6209.107300] workqueue events_power_efficient: flags=0x80 [ 6209.117449] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6209.122999] pending: neigh_periodic_work, check_lifetime [ 6209.126370] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6209.135112] pending: do_cache_clean, neigh_periodic_work [ 6209.138433] workqueue mm_percpu_wq: flags=0x8 [ 6209.141407] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6209.148168] pending: lru_add_drain_per_cpu [ 6209.152053] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6209.158180] pending: vmstat_update, lru_add_drain_per_cpu [ 6209.162495] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6209.167216] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6209.173017] workqueue writeback: flags=0x4a [ 6209.175779] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6209.189412] pending: wb_workfn, wb_workfn [ 6209.198345] workqueue xprtiod: flags=0xa [ 6209.202343] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6209.205467] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6209.210929] workqueue ipv6_addrconf: flags=0x40008 [ 6209.212543] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6209.220919] pending: addrconf_verify_work [ 6209.224969] workqueue knbd0-recv: flags=0x1a [ 6209.226514] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6209.240486] in-flight: 655:recv_work [ 6209.242014] workqueue ptlrpc_pinger: flags=0x2 [ 6209.246649] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6209.257154] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6209.262119] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4184s workers=4 idle: 36396 38 11188 [ 6209.267931] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6199s workers=2 idle: 658 [ 6212.052018] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6212.056997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6212.114853] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6212.134051] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6212.140989] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6212.145415] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6212.168082] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6212.172605] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6212.178368] RDX: 0000000000000833 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6212.185033] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6212.196443] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6212.204356] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6212.219325] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6212.227313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6212.230163] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6212.232734] Call Trace: [ 6212.233494] [ 6212.234135] ? show_regs.cold.9+0x22/0x2f [ 6212.235387] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6212.236924] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6212.238390] ? watchdog+0x40/0x40 [ 6212.239443] ? hrtimer_interrupt+0x128/0x2c0 [ 6212.240753] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6212.242304] ? apic_timer_interrupt+0xf/0x20 [ 6212.243636] [ 6212.244327] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6212.245958] do_raw_spin_lock+0x177/0x190 [ 6212.247127] _raw_spin_lock+0x21/0x30 [ 6212.248212] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6212.250193] ? do_raw_spin_unlock+0x75/0x190 [ 6212.251740] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6212.253731] ? do_wait_intr+0xf0/0xf0 [ 6212.254824] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6212.256782] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6212.258506] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6212.260460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6212.262049] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6212.263484] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6212.264738] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6212.265854] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6212.267429] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6212.269399] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6212.271219] lmv_intent_open+0x338/0x1150 [lmv] [ 6212.283439] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6212.290602] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6212.292994] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6212.295019] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6212.296805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6212.298993] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6212.300560] ? do_raw_spin_unlock+0x75/0x190 [ 6212.302119] ? _raw_spin_unlock+0x12/0x30 [ 6212.303747] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6212.305948] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6212.307605] ll_atomic_open+0x36a/0xd70 [lustre] [ 6212.309181] lookup_open+0x356/0xb40 [ 6212.310499] do_last+0x313/0xfc0 [ 6212.311412] ? nd_jump_root+0xe5/0x160 [ 6212.312897] ? path_init+0x437/0x520 [ 6212.314366] path_openat+0xf7/0x500 [ 6212.315660] ? do_raw_spin_unlock+0x75/0x190 [ 6212.317037] do_filp_open+0x99/0x140 [ 6212.318199] ? getname_flags+0x6e/0x330 [ 6212.319473] ? __check_object_size+0xff/0x256 [ 6212.320766] ? do_raw_spin_unlock+0x75/0x190 [ 6212.322430] ? _raw_spin_unlock+0x12/0x30 [ 6212.327430] do_sys_openat2+0x2b4/0x410 [ 6212.328536] do_sys_open+0x73/0xa0 [ 6212.329891] __x64_sys_openat+0x24/0x30 [ 6212.331656] do_syscall_64+0xc1/0x440 [ 6212.335173] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6212.336965] RIP: 0033:0x7fdc0afbf332 [ 6212.338194] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6212.344920] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6212.347051] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6212.351743] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6212.354156] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6212.356550] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6212.361165] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6219.437487] rcu: INFO: rcu_sched self-detected stall on CPU [ 6219.440440] rcu: 0-...!: (2516417 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 6219.447793] (t=4200094 jiffies g=270761 q=3682) [ 6219.449951] rcu: rcu_sched kthread starved for 4195026 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 6219.454403] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6219.458154] rcu: RCU grace-period kthread stack dump: [ 6219.460824] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6219.464786] Call Trace: [ 6219.466361] __schedule+0x351/0xcb0 [ 6219.470598] schedule+0xc0/0x180 [ 6219.473771] schedule_timeout+0xb4/0x190 [ 6219.475739] ? do_raw_spin_unlock+0x75/0x190 [ 6219.478172] ? __next_timer_interrupt+0x160/0x160 [ 6219.480250] ? prepare_to_swait_event+0xf3/0x140 [ 6219.485805] rcu_gp_kthread+0x8dd/0x15b0 [ 6219.494163] ? force_qs_rnp+0x2c0/0x2c0 [ 6219.498444] kthread+0x1d1/0x200 [ 6219.505568] ? set_kthread_struct+0x70/0x70 [ 6219.508718] ret_from_fork+0x1f/0x30 [ 6219.512727] rcu: Stack dump where RCU GP kthread last ran: [ 6219.522612] Sending NMI from CPU 0 to CPUs 3: [ 6219.537415] NMI backtrace for cpu 3 [ 6219.537422] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6219.537424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6219.537426] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6219.537428] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6219.537430] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000216 [ 6219.537432] RAX: 0000000000006014 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6219.537433] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6219.537435] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6219.537436] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6219.537438] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6219.537439] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6219.537441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6219.537442] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6219.537443] Call Trace: [ 6219.537444] [ 6219.537446] ? show_regs.cold.9+0x22/0x2f [ 6219.537446] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6219.537448] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6219.537449] ? nmi_handle+0xbd/0x210 [ 6219.537450] ? default_do_nmi+0x4b/0x1b0 [ 6219.537451] ? do_nmi+0x280/0x380 [ 6219.537452] ? end_repeat_nmi+0x16/0x1f [ 6219.537453] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6219.537454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6219.537455] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6219.537456] [ 6219.537457] do_raw_spin_lock+0x177/0x190 [ 6219.537458] _raw_spin_lock+0x21/0x30 [ 6219.537459] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6219.537460] ? do_raw_spin_unlock+0x75/0x190 [ 6219.537461] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6219.537463] ? do_wait_intr+0xf0/0xf0 [ 6219.537463] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6219.537465] mdc_close+0x53b/0xd70 [mdc] [ 6219.537466] lmv_close+0x18d/0x490 [lmv] [ 6219.537467] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6219.537468] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6219.537470] ll_md_close+0x274/0x800 [lustre] [ 6219.537471] ll_file_release+0x128/0xc50 [lustre] [ 6219.537472] __fput+0x10d/0x440 [ 6219.537473] ____fput+0x12/0x20 [ 6219.537474] task_work_run+0xc6/0x110 [ 6219.537476] exit_to_usermode_loop+0x1dd/0x1f0 [ 6219.537477] do_syscall_64+0x3d6/0x440 [ 6219.537478] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6219.537479] RIP: 0033:0x7fdc0afbfcd8 [ 6219.537481] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6219.537483] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6219.537486] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6219.537488] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6219.537489] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6219.537491] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6219.537492] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6219.538411] NMI backtrace for cpu 0 [ 6219.768467] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6219.779409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6219.782448] Call Trace: [ 6219.783412] [ 6219.784231] ? dump_stack+0xbb/0x10e [ 6219.785756] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6219.787443] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6219.789168] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6219.792507] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6219.797288] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6219.801205] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6219.802716] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6219.805524] ? update_process_times+0x6d/0xc0 [ 6219.809059] ? tick_sched_handle.isra.18+0x27/0x90 [ 6219.812372] ? tick_sched_timer+0x53/0xb0 [ 6219.813879] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6219.816753] ? tick_sched_handle.isra.18+0x90/0x90 [ 6219.821420] ? hrtimer_interrupt+0x128/0x2c0 [ 6219.836003] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6219.840425] ? apic_timer_interrupt+0xf/0x20 [ 6219.843935] [ 6219.844863] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6219.848244] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6219.852169] ? dump_stack+0xf6/0x10e [ 6219.855236] ? do_raw_spin_lock+0x177/0x190 [ 6219.856812] ? _raw_spin_lock+0x21/0x30 [ 6219.859759] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6219.861834] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6219.864460] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6219.865644] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6219.869191] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6219.871285] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6219.873353] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6219.876268] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6219.878239] ? do_wait_intr+0xf0/0xf0 [ 6219.884112] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6219.889682] ? kthread+0x1d1/0x200 [ 6219.891642] ? set_kthread_struct+0x70/0x70 [ 6219.894272] ? ret_from_fork+0x1f/0x30 [ 6220.051030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6220.056771] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6220.084896] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6220.092060] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6220.103874] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6220.105844] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6220.111475] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6220.119313] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6220.125155] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6220.129534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6220.135327] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6220.138239] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6220.141931] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6220.145745] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6220.148660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6220.151052] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6220.153794] Call Trace: [ 6220.154866] [ 6220.155666] ? show_regs.cold.9+0x22/0x2f [ 6220.157272] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6220.159630] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6220.161405] ? watchdog+0x40/0x40 [ 6220.164222] ? hrtimer_interrupt+0x128/0x2c0 [ 6220.165609] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6220.169253] ? apic_timer_interrupt+0xf/0x20 [ 6220.174221] [ 6220.175411] ? native_safe_halt+0x2e/0x50 [ 6220.177472] kvm_wait+0xcd/0xe0 [ 6220.178903] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6220.181237] do_raw_spin_lock+0x177/0x190 [ 6220.182872] _raw_spin_lock+0x21/0x30 [ 6220.184628] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6220.193896] process_one_work+0x2c8/0x700 [ 6220.196586] worker_thread+0x296/0x6e0 [ 6220.197835] ? rescuer_thread+0x570/0x570 [ 6220.202814] kthread+0x1d1/0x200 [ 6220.204259] ? set_kthread_struct+0x70/0x70 [ 6220.206123] ret_from_fork+0x1f/0x30 [ 6239.712656] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4214s! [ 6239.718976] Showing busy workqueues and worker pools: [ 6239.720960] workqueue events: flags=0x0 [ 6239.723058] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6239.726515] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6239.730102] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6239.733477] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6239.741888] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6239.745642] pending: kfree_rcu_monitor [ 6239.748010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6239.751361] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6239.758234] workqueue events_long: flags=0x0 [ 6239.760661] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6239.765862] pending: lu_objects_flush [obdclass] [ 6239.769028] workqueue events_unbound: flags=0x2 [ 6239.771082] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6239.774694] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6239.780822] workqueue events_power_efficient: flags=0x80 [ 6239.783862] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6239.787576] pending: neigh_periodic_work, check_lifetime [ 6239.792876] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6239.797099] pending: do_cache_clean, neigh_periodic_work [ 6239.800198] workqueue mm_percpu_wq: flags=0x8 [ 6239.803358] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6239.806655] pending: lru_add_drain_per_cpu [ 6239.808279] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6239.811176] pending: vmstat_update, lru_add_drain_per_cpu [ 6239.814054] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6239.817401] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6239.821390] workqueue writeback: flags=0x4a [ 6239.823585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6239.827174] pending: wb_workfn, wb_workfn [ 6239.829307] workqueue xprtiod: flags=0xa [ 6239.831159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6239.834469] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6239.837477] workqueue ipv6_addrconf: flags=0x40008 [ 6239.839157] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6239.842964] pending: addrconf_verify_work [ 6239.845298] workqueue knbd0-recv: flags=0x1a [ 6239.847561] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6239.851657] in-flight: 655:recv_work [ 6239.853980] workqueue ptlrpc_pinger: flags=0x2 [ 6239.856961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6239.862518] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6239.869041] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4214s workers=4 idle: 36396 38 11188 [ 6239.874452] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6230s workers=2 idle: 658 [ 6240.052020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6240.055421] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6240.077835] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6240.081996] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6240.085671] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6240.088509] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6240.096430] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6240.099659] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6240.103184] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6240.106395] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6240.109569] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6240.112644] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6240.116554] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6240.121435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6240.125637] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6240.130228] Call Trace: [ 6240.131590] [ 6240.133633] ? show_regs.cold.9+0x22/0x2f [ 6240.140327] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6240.144496] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6240.148296] ? watchdog+0x40/0x40 [ 6240.154046] ? hrtimer_interrupt+0x128/0x2c0 [ 6240.158238] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6240.162636] ? apic_timer_interrupt+0xf/0x20 [ 6240.165810] [ 6240.167119] ? native_safe_halt+0x2e/0x50 [ 6240.170252] kvm_wait+0xcd/0xe0 [ 6240.172161] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6240.175843] do_raw_spin_lock+0x177/0x190 [ 6240.178418] _raw_spin_lock+0x21/0x30 [ 6240.181173] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6240.184515] ? do_raw_spin_unlock+0x75/0x190 [ 6240.186285] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6240.188977] ? do_wait_intr+0xf0/0xf0 [ 6240.190518] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6240.193364] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6240.196743] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6240.199916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6240.202134] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6240.205328] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6240.207932] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6240.210713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6240.214025] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6240.217826] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6240.220441] lmv_intent_open+0x338/0x1150 [lmv] [ 6240.222684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6240.225577] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6240.228586] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6240.230982] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6240.233641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6240.236505] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6240.238823] ? do_raw_spin_unlock+0x75/0x190 [ 6240.241250] ? _raw_spin_unlock+0x12/0x30 [ 6240.242969] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6240.246432] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6240.249367] ll_atomic_open+0x36a/0xd70 [lustre] [ 6240.251879] lookup_open+0x356/0xb40 [ 6240.255248] do_last+0x313/0xfc0 [ 6240.256868] ? nd_jump_root+0xe5/0x160 [ 6240.259084] ? path_init+0x437/0x520 [ 6240.261184] path_openat+0xf7/0x500 [ 6240.263276] ? do_raw_spin_unlock+0x75/0x190 [ 6240.265544] do_filp_open+0x99/0x140 [ 6240.266927] ? getname_flags+0x6e/0x330 [ 6240.268318] ? __check_object_size+0xff/0x256 [ 6240.270681] ? do_raw_spin_unlock+0x75/0x190 [ 6240.273092] ? _raw_spin_unlock+0x12/0x30 [ 6240.275331] do_sys_openat2+0x2b4/0x410 [ 6240.277369] do_sys_open+0x73/0xa0 [ 6240.279131] __x64_sys_openat+0x24/0x30 [ 6240.281179] do_syscall_64+0xc1/0x440 [ 6240.283124] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6240.285622] RIP: 0033:0x7fdc0afbf332 [ 6240.287619] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6240.297299] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6240.301321] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6240.305117] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6240.308647] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6240.312368] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6240.316274] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6244.054007] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 6244.058328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6244.091548] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6244.100417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6244.104702] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6244.108514] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6244.118860] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6244.122853] RAX: 0000000000000c29 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6244.126275] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6244.128547] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6244.131486] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6244.134484] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6244.136696] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6244.139288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6244.142251] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6244.145489] Call Trace: [ 6244.146557] [ 6244.147383] ? show_regs.cold.9+0x22/0x2f [ 6244.148804] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6244.150356] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6244.151975] ? watchdog+0x40/0x40 [ 6244.152890] ? hrtimer_interrupt+0x128/0x2c0 [ 6244.154800] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6244.157078] ? apic_timer_interrupt+0xf/0x20 [ 6244.158814] [ 6244.159526] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6244.161617] do_raw_spin_lock+0x177/0x190 [ 6244.163244] _raw_spin_lock+0x21/0x30 [ 6244.164838] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6244.167437] ? do_raw_spin_unlock+0x75/0x190 [ 6244.169388] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6244.172445] ? do_wait_intr+0xf0/0xf0 [ 6244.174245] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6244.176807] mdc_close+0x53b/0xd70 [mdc] [ 6244.178704] lmv_close+0x18d/0x490 [lmv] [ 6244.180452] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6244.183292] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6244.185622] ll_md_close+0x274/0x800 [lustre] [ 6244.188011] ll_file_release+0x128/0xc50 [lustre] [ 6244.189844] __fput+0x10d/0x440 [ 6244.191341] ____fput+0x12/0x20 [ 6244.192860] task_work_run+0xc6/0x110 [ 6244.194582] exit_to_usermode_loop+0x1dd/0x1f0 [ 6244.196630] do_syscall_64+0x3d6/0x440 [ 6244.198383] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6244.200784] RIP: 0033:0x7fdc0afbfcd8 [ 6244.202425] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6244.210804] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6244.214067] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6244.217344] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6244.220784] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6244.224087] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6244.227328] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6248.051354] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6248.053762] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6248.089768] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6248.097273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6248.107638] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6248.112685] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6248.116231] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6248.134124] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6248.136441] RAX: 000000000000595b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6248.142362] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6248.148421] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6248.153329] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6248.160455] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6248.162837] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6248.167918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6248.173048] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6248.178029] Call Trace: [ 6248.179910] [ 6248.180585] ? show_regs.cold.9+0x22/0x2f [ 6248.183473] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6248.190592] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6248.192806] ? watchdog+0x40/0x40 [ 6248.194335] ? hrtimer_interrupt+0x128/0x2c0 [ 6248.196972] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6248.201697] ? apic_timer_interrupt+0xf/0x20 [ 6248.207203] [ 6248.207963] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6248.211844] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6248.214451] do_raw_spin_lock+0x177/0x190 [ 6248.217382] _raw_spin_lock+0x21/0x30 [ 6248.218938] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6248.221652] process_one_work+0x2c8/0x700 [ 6248.223349] worker_thread+0x296/0x6e0 [ 6248.228611] ? rescuer_thread+0x570/0x570 [ 6248.229916] kthread+0x1d1/0x200 [ 6248.232262] ? set_kthread_struct+0x70/0x70 [ 6248.233949] ret_from_fork+0x1f/0x30 [ 6264.048021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6264.050406] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6264.065732] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6264.069411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6264.072238] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6264.073713] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6264.080289] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6264.083037] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6264.085629] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6264.088126] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6264.090928] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6264.094266] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6264.097149] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6264.099834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6264.101688] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6264.104149] Call Trace: [ 6264.105267] [ 6264.106329] ? show_regs.cold.9+0x22/0x2f [ 6264.107490] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6264.109367] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6264.110911] ? watchdog+0x40/0x40 [ 6264.112284] ? hrtimer_interrupt+0x128/0x2c0 [ 6264.113561] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6264.115290] ? apic_timer_interrupt+0xf/0x20 [ 6264.116624] [ 6264.117519] ? native_safe_halt+0x2e/0x50 [ 6264.118905] kvm_wait+0xcd/0xe0 [ 6264.119940] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6264.121683] ? dump_stack+0xf6/0x10e [ 6264.123047] do_raw_spin_lock+0x177/0x190 [ 6264.124710] _raw_spin_lock+0x21/0x30 [ 6264.126120] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6264.128144] obd_mod_free+0x46/0x140 [ptlrpc] [ 6264.129869] mdc_commit_open+0x91/0xb0 [mdc] [ 6264.131500] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6264.133568] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6264.135728] after_reply+0xe2c/0x1780 [ptlrpc] [ 6264.137668] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6264.139884] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6264.141574] ? do_wait_intr+0xf0/0xf0 [ 6264.142763] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6264.144666] kthread+0x1d1/0x200 [ 6264.146279] ? set_kthread_struct+0x70/0x70 [ 6264.147720] ret_from_fork+0x1f/0x30 [ 6268.052152] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6268.057108] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6268.076337] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6268.080942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6268.084594] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6268.087212] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6268.095280] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6268.098630] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6268.101496] RDX: 0000000000005fcc RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6268.104374] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6268.107366] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6268.110911] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6268.116440] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6268.119910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6268.122248] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6268.125047] Call Trace: [ 6268.126029] [ 6268.127023] ? show_regs.cold.9+0x22/0x2f [ 6268.128995] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6268.131133] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6268.133366] ? watchdog+0x40/0x40 [ 6268.135214] ? hrtimer_interrupt+0x128/0x2c0 [ 6268.137460] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6268.140155] ? apic_timer_interrupt+0xf/0x20 [ 6268.142431] [ 6268.144254] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6268.146558] do_raw_spin_lock+0x177/0x190 [ 6268.148070] _raw_spin_lock+0x21/0x30 [ 6268.149273] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6268.151834] ? do_raw_spin_unlock+0x75/0x190 [ 6268.153741] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6268.156691] ? do_wait_intr+0xf0/0xf0 [ 6268.158221] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6268.161041] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6268.164329] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6268.168712] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6268.171388] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6268.174119] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6268.176175] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6268.179153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6268.180968] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6268.183563] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6268.185500] lmv_intent_open+0x338/0x1150 [lmv] [ 6268.187988] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6268.190254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6268.192267] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6268.193945] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6268.196380] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6268.198256] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6268.199958] ? do_raw_spin_unlock+0x75/0x190 [ 6268.201205] ? _raw_spin_unlock+0x12/0x30 [ 6268.202610] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6268.204443] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6268.206371] ll_atomic_open+0x36a/0xd70 [lustre] [ 6268.208244] lookup_open+0x356/0xb40 [ 6268.209447] do_last+0x313/0xfc0 [ 6268.210469] ? nd_jump_root+0xe5/0x160 [ 6268.211956] ? path_init+0x437/0x520 [ 6268.212989] path_openat+0xf7/0x500 [ 6268.214130] ? do_raw_spin_unlock+0x75/0x190 [ 6268.215482] do_filp_open+0x99/0x140 [ 6268.216909] ? getname_flags+0x6e/0x330 [ 6268.218105] ? __check_object_size+0xff/0x256 [ 6268.219800] ? do_raw_spin_unlock+0x75/0x190 [ 6268.221162] ? _raw_spin_unlock+0x12/0x30 [ 6268.222677] do_sys_openat2+0x2b4/0x410 [ 6268.224199] do_sys_open+0x73/0xa0 [ 6268.225522] __x64_sys_openat+0x24/0x30 [ 6268.227071] do_syscall_64+0xc1/0x440 [ 6268.228498] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6268.230352] RIP: 0033:0x7fdc0afbf332 [ 6268.231762] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6268.239764] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6268.242997] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6268.245461] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6268.247742] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6268.249534] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6268.252066] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6269.919096] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4244s! [ 6269.924159] Showing busy workqueues and worker pools: [ 6269.926725] workqueue events: flags=0x0 [ 6269.928488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6269.932157] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6269.934913] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6269.940981] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6269.947199] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6269.952418] pending: kfree_rcu_monitor [ 6269.954809] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6269.958646] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6269.966721] workqueue events_long: flags=0x0 [ 6269.969326] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6269.972820] pending: lu_objects_flush [obdclass] [ 6269.976123] workqueue events_unbound: flags=0x2 [ 6269.979356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6269.983074] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6269.989041] workqueue events_power_efficient: flags=0x80 [ 6269.992932] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6269.997974] pending: neigh_periodic_work, check_lifetime [ 6270.002307] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6270.006426] pending: do_cache_clean, neigh_periodic_work [ 6270.009691] workqueue mm_percpu_wq: flags=0x8 [ 6270.012602] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6270.015634] pending: lru_add_drain_per_cpu [ 6270.017660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6270.022097] pending: vmstat_update, lru_add_drain_per_cpu [ 6270.023914] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6270.027383] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6270.030644] workqueue writeback: flags=0x4a [ 6270.032607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6270.035351] pending: wb_workfn, wb_workfn [ 6270.036854] workqueue xprtiod: flags=0xa [ 6270.038445] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6270.040269] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6270.042428] workqueue ipv6_addrconf: flags=0x40008 [ 6270.044504] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6270.052646] pending: addrconf_verify_work [ 6270.056390] workqueue knbd0-recv: flags=0x1a [ 6270.059743] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6270.062458] in-flight: 655:recv_work [ 6270.064499] workqueue ptlrpc_pinger: flags=0x2 [ 6270.066406] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6270.068408] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6270.073327] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4245s workers=4 idle: 36396 38 11188 [ 6270.078063] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6260s workers=2 idle: 658 [ 6272.054692] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 6272.060530] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6272.092715] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6272.102181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6272.107757] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6272.115939] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6272.137520] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6272.145783] RAX: 0000000000007dd7 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6272.154511] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6272.162586] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6272.171095] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6272.174754] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6272.182753] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6272.189026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6272.192085] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6272.195873] Call Trace: [ 6272.197497] [ 6272.198270] ? show_regs.cold.9+0x22/0x2f [ 6272.201993] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6272.204623] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6272.207862] ? watchdog+0x40/0x40 [ 6272.210018] ? hrtimer_interrupt+0x128/0x2c0 [ 6272.212064] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6272.213831] ? apic_timer_interrupt+0xf/0x20 [ 6272.215502] [ 6272.216701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6272.219246] do_raw_spin_lock+0x177/0x190 [ 6272.220948] _raw_spin_lock+0x21/0x30 [ 6272.222681] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6272.224869] ? do_raw_spin_unlock+0x75/0x190 [ 6272.226870] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6272.229696] ? do_wait_intr+0xf0/0xf0 [ 6272.231227] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6272.235091] mdc_close+0x53b/0xd70 [mdc] [ 6272.237398] lmv_close+0x18d/0x490 [lmv] [ 6272.239185] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6272.241749] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6272.244121] ll_md_close+0x274/0x800 [lustre] [ 6272.245957] ll_file_release+0x128/0xc50 [lustre] [ 6272.248132] __fput+0x10d/0x440 [ 6272.249324] ____fput+0x12/0x20 [ 6272.250989] task_work_run+0xc6/0x110 [ 6272.252789] exit_to_usermode_loop+0x1dd/0x1f0 [ 6272.254977] do_syscall_64+0x3d6/0x440 [ 6272.257135] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6272.259539] RIP: 0033:0x7fdc0afbfcd8 [ 6272.261848] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6272.271005] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6272.274726] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6272.278176] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6272.281824] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6272.285568] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6272.288769] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6276.051048] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6276.054462] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6276.081287] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6276.087469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6276.092449] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6276.096334] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6276.098398] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6276.108071] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6276.112422] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6276.115599] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6276.119106] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6276.122378] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6276.125325] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6276.129635] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6276.133697] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6276.135648] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6276.138554] Call Trace: [ 6276.139714] [ 6276.140874] ? show_regs.cold.9+0x22/0x2f [ 6276.142922] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6276.145298] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6276.147503] ? watchdog+0x40/0x40 [ 6276.149229] ? hrtimer_interrupt+0x128/0x2c0 [ 6276.151278] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6276.153306] ? apic_timer_interrupt+0xf/0x20 [ 6276.155196] [ 6276.157086] ? native_safe_halt+0x2e/0x50 [ 6276.158412] kvm_wait+0xcd/0xe0 [ 6276.159615] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6276.161464] do_raw_spin_lock+0x177/0x190 [ 6276.163221] _raw_spin_lock+0x21/0x30 [ 6276.165115] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6276.168034] process_one_work+0x2c8/0x700 [ 6276.169614] worker_thread+0x296/0x6e0 [ 6276.172726] ? rescuer_thread+0x570/0x570 [ 6276.174446] kthread+0x1d1/0x200 [ 6276.176219] ? set_kthread_struct+0x70/0x70 [ 6276.178285] ret_from_fork+0x1f/0x30 [ 6296.048177] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6296.053214] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6296.053531] Modules linked in: [ 6296.057532] Modules linked in: [ 6296.057535] lustre(O) osp(O) ofd(O) [ 6296.059385] lustre(O) [ 6296.060515] lod(O) [ 6296.063187] osp(O) [ 6296.064738] mdt(O) [ 6296.065695] ofd(O) [ 6296.066734] mdd(O) [ 6296.067734] lod(O) [ 6296.068725] mgs(O) [ 6296.069632] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [ 6296.070531] lquota(O) [ 6296.071542] intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6296.071576] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6296.071581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6296.071584] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6296.071599] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6296.071604] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6296.071608] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6296.071610] RDX: 0000000000007625 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6296.071613] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6296.080978] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs [ 6296.082314] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6296.092146] ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6296.092177] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6296.092185] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6296.092188] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6296.092203] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6296.092207] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6296.092270] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6296.092274] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6296.092276] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6296.092278] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6296.092280] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6296.092284] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6296.092286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6296.092289] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6296.092298] Call Trace: [ 6296.092305] [ 6296.092317] ? show_regs.cold.9+0x22/0x2f [ 6296.092329] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6296.092335] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6296.092340] ? watchdog+0x40/0x40 [ 6296.092345] ? hrtimer_interrupt+0x128/0x2c0 [ 6296.092353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6296.092364] ? apic_timer_interrupt+0xf/0x20 [ 6296.092369] [ 6296.092372] ? native_safe_halt+0x2e/0x50 [ 6296.092376] kvm_wait+0xcd/0xe0 [ 6296.092384] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6296.092391] ? dump_stack+0xf6/0x10e [ 6296.092396] do_raw_spin_lock+0x177/0x190 [ 6296.092401] _raw_spin_lock+0x21/0x30 [ 6296.092408] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6296.092820] obd_mod_free+0x46/0x140 [ptlrpc] [ 6296.093018] mdc_commit_open+0x91/0xb0 [mdc] [ 6296.093062] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6296.093432] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6296.093647] after_reply+0xe2c/0x1780 [ptlrpc] [ 6296.093863] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6296.094070] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6296.094374] ? do_wait_intr+0xf0/0xf0 [ 6296.094385] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6296.094730] kthread+0x1d1/0x200 [ 6296.094736] ? set_kthread_struct+0x70/0x70 [ 6296.094739] ret_from_fork+0x1f/0x30 [ 6296.401247] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6296.403769] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6296.407839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6296.411792] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6296.414946] Call Trace: [ 6296.416472] [ 6296.417512] ? show_regs.cold.9+0x22/0x2f [ 6296.418698] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6296.422438] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6296.423898] ? watchdog+0x40/0x40 [ 6296.425599] ? hrtimer_interrupt+0x128/0x2c0 [ 6296.429535] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6296.432632] ? apic_timer_interrupt+0xf/0x20 [ 6296.435734] [ 6296.437789] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6296.445844] do_raw_spin_lock+0x177/0x190 [ 6296.450505] _raw_spin_lock+0x21/0x30 [ 6296.454533] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6296.459509] ? do_raw_spin_unlock+0x75/0x190 [ 6296.461745] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6296.464450] ? do_wait_intr+0xf0/0xf0 [ 6296.465835] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6296.469862] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6296.473352] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6296.475896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6296.478523] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6296.480576] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6296.482759] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6296.484583] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6296.486681] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6296.489471] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6296.491856] lmv_intent_open+0x338/0x1150 [lmv] [ 6296.493873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6296.496445] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6296.498649] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6296.500590] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6296.502276] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6296.504453] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6296.506266] ? do_raw_spin_unlock+0x75/0x190 [ 6296.508029] ? _raw_spin_unlock+0x12/0x30 [ 6296.509792] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6296.512848] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6296.515883] ll_atomic_open+0x36a/0xd70 [lustre] [ 6296.519721] lookup_open+0x356/0xb40 [ 6296.521043] do_last+0x313/0xfc0 [ 6296.523666] ? nd_jump_root+0xe5/0x160 [ 6296.527475] ? path_init+0x437/0x520 [ 6296.531548] path_openat+0xf7/0x500 [ 6296.534429] ? do_raw_spin_unlock+0x75/0x190 [ 6296.536395] do_filp_open+0x99/0x140 [ 6296.537822] ? getname_flags+0x6e/0x330 [ 6296.546735] ? __check_object_size+0xff/0x256 [ 6296.551069] ? do_raw_spin_unlock+0x75/0x190 [ 6296.554655] ? _raw_spin_unlock+0x12/0x30 [ 6296.556078] do_sys_openat2+0x2b4/0x410 [ 6296.557460] do_sys_open+0x73/0xa0 [ 6296.562659] __x64_sys_openat+0x24/0x30 [ 6296.565519] do_syscall_64+0xc1/0x440 [ 6296.567469] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6296.570413] RIP: 0033:0x7fdc0afbf332 [ 6296.572711] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6296.598539] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6296.610619] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6296.619143] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6296.626889] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6296.631383] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6296.641380] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6300.054017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6300.057571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6300.079672] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6300.085294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6300.089728] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6300.092847] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6300.103032] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6300.115959] RAX: 0000000000001b8b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6300.118689] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6300.122019] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6300.126061] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6300.128040] BUG: workqueue lockup - pool [ 6300.132705] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6300.132716] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6300.132719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6300.132721] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6300.132735] Call Trace: [ 6300.132742] [ 6300.132746] ? show_regs.cold.9+0x22/0x2f [ 6300.132758] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6300.132764] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6300.132769] ? watchdog+0x40/0x40 [ 6300.132773] ? hrtimer_interrupt+0x128/0x2c0 [ 6300.132777] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6300.132785] ? apic_timer_interrupt+0xf/0x20 [ 6300.132789] [ 6300.132792] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6300.132803] do_raw_spin_lock+0x177/0x190 [ 6300.132808] _raw_spin_lock+0x21/0x30 [ 6300.132813] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6300.133098] ? do_raw_spin_unlock+0x75/0x190 [ 6300.133103] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6300.137598] cpus=0-3 flags=0x4 nice=0 stuck for 4275s! [ 6300.137620] Showing busy workqueues and worker pools: [ 6300.137622] workqueue events: flags=0x0 [ 6300.137626] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6300.137635] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6300.137918] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6300.137927] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6300.138481] pwq 2: [ 6300.140822] ? do_wait_intr+0xf0/0xf0 [ 6300.146982] cpus=1 [ 6300.149094] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6300.157017] node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6300.157035] pending: kfree_rcu_monitor [ 6300.157070] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6300.157082] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc] [ 6300.158725] mdc_close+0x53b/0xd70 [mdc] [ 6300.158770] lmv_close+0x18d/0x490 [lmv] [ 6300.158801] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6300.158910] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6300.158992] ll_md_close+0x274/0x800 [lustre] [ 6300.159076] ll_file_release+0x128/0xc50 [lustre] [ 6300.159156] __fput+0x10d/0x440 [ 6300.159165] ____fput+0x12/0x20 [ 6300.159169] task_work_run+0xc6/0x110 [ 6300.159176] exit_to_usermode_loop+0x1dd/0x1f0 [ 6300.159183] do_syscall_64+0x3d6/0x440 [ 6300.159188] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6300.159197] RIP: 0033:0x7fdc0afbfcd8 [ 6300.159203] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6300.159206] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6300.159210] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6300.159211] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6300.159213] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6300.159214] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6300.159216] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6300.316076] , sec_gc_main [ptlrpc] [ 6300.318097] workqueue events_long: flags=0x0 [ 6300.321949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6300.325752] pending: lu_objects_flush [obdclass] [ 6300.327156] workqueue events_unbound: flags=0x2 [ 6300.337831] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6300.341521] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6300.354036] workqueue events_power_efficient: flags=0x80 [ 6300.356991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6300.360399] pending: neigh_periodic_work, check_lifetime [ 6300.362943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6300.365997] pending: do_cache_clean, neigh_periodic_work [ 6300.368982] workqueue mm_percpu_wq: flags=0x8 [ 6300.370338] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6300.378083] pending: lru_add_drain_per_cpu [ 6300.382565] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6300.385938] pending: vmstat_update, lru_add_drain_per_cpu [ 6300.392884] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6300.398708] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6300.401275] workqueue writeback: flags=0x4a [ 6300.404320] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6300.406175] pending: wb_workfn, wb_workfn [ 6300.407489] workqueue xprtiod: flags=0xa [ 6300.419045] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6300.427520] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6300.432030] workqueue ipv6_addrconf: flags=0x40008 [ 6300.433854] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6300.438947] pending: addrconf_verify_work [ 6300.441800] workqueue knbd0-recv: flags=0x1a [ 6300.444291] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6300.456589] in-flight: 655:recv_work [ 6300.457891] workqueue ptlrpc_pinger: flags=0x2 [ 6300.460345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6300.466862] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6300.472019] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4275s workers=4 idle: 36396 38 11188 [ 6300.475711] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6291s workers=2 idle: 658 [ 6304.051095] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6304.054439] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6304.075643] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6304.081004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6304.084879] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6304.088145] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6304.090547] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6304.099061] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6304.102223] RAX: 000000000000321e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6304.105272] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6304.108847] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6304.113168] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6304.116399] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6304.120155] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6304.124008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6304.126515] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6304.129837] Call Trace: [ 6304.130912] [ 6304.131970] ? show_regs.cold.9+0x22/0x2f [ 6304.133796] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6304.135646] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6304.137536] ? watchdog+0x40/0x40 [ 6304.139200] ? hrtimer_interrupt+0x128/0x2c0 [ 6304.141023] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6304.143019] ? apic_timer_interrupt+0xf/0x20 [ 6304.144818] [ 6304.145787] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6304.148055] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6304.150447] do_raw_spin_lock+0x177/0x190 [ 6304.152144] _raw_spin_lock+0x21/0x30 [ 6304.153881] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6304.156033] process_one_work+0x2c8/0x700 [ 6304.157646] worker_thread+0x296/0x6e0 [ 6304.159115] ? rescuer_thread+0x570/0x570 [ 6304.160378] kthread+0x1d1/0x200 [ 6304.161947] ? set_kthread_struct+0x70/0x70 [ 6304.163408] ret_from_fork+0x1f/0x30 [ 6324.052518] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6324.071275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6324.120456] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6324.126421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6324.130843] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6324.132931] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6324.143435] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6324.145859] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6324.148158] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6324.150095] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6324.152369] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6324.154525] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6324.156659] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6324.159206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6324.161053] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6324.167949] Call Trace: [ 6324.169042] [ 6324.169835] ? show_regs.cold.9+0x22/0x2f [ 6324.174974] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6324.180188] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6324.182630] ? watchdog+0x40/0x40 [ 6324.183887] ? hrtimer_interrupt+0x128/0x2c0 [ 6324.185419] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6324.190824] ? apic_timer_interrupt+0xf/0x20 [ 6324.195322] [ 6324.196875] ? native_safe_halt+0x2e/0x50 [ 6324.199932] kvm_wait+0xcd/0xe0 [ 6324.202090] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6324.205610] do_raw_spin_lock+0x177/0x190 [ 6324.207456] _raw_spin_lock+0x21/0x30 [ 6324.209112] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6324.211973] ? do_raw_spin_unlock+0x75/0x190 [ 6324.215714] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6324.223497] ? do_wait_intr+0xf0/0xf0 [ 6324.225089] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6324.227890] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6324.232036] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6324.234968] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6324.237061] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6324.245484] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6324.250566] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6324.251745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6324.255223] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6324.261074] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6324.264764] lmv_intent_open+0x338/0x1150 [lmv] [ 6324.265963] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6324.268309] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6324.270496] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6324.272324] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6324.274335] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6324.276548] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6324.278539] ? do_raw_spin_unlock+0x75/0x190 [ 6324.280108] ? _raw_spin_unlock+0x12/0x30 [ 6324.281654] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6324.283812] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6324.285951] ll_atomic_open+0x36a/0xd70 [lustre] [ 6324.287962] lookup_open+0x356/0xb40 [ 6324.289415] do_last+0x313/0xfc0 [ 6324.290796] ? nd_jump_root+0xe5/0x160 [ 6324.292399] ? path_init+0x437/0x520 [ 6324.293651] path_openat+0xf7/0x500 [ 6324.295068] ? do_raw_spin_unlock+0x75/0x190 [ 6324.296901] do_filp_open+0x99/0x140 [ 6324.298349] ? getname_flags+0x6e/0x330 [ 6324.299822] ? __check_object_size+0xff/0x256 [ 6324.301837] ? do_raw_spin_unlock+0x75/0x190 [ 6324.303961] ? _raw_spin_unlock+0x12/0x30 [ 6324.305692] do_sys_openat2+0x2b4/0x410 [ 6324.307435] do_sys_open+0x73/0xa0 [ 6324.308662] __x64_sys_openat+0x24/0x30 [ 6324.310571] do_syscall_64+0xc1/0x440 [ 6324.312324] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6324.314410] RIP: 0033:0x7fdc0afbf332 [ 6324.315935] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6324.324334] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6324.327789] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6324.333688] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6324.337173] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6324.340325] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6324.343071] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6328.048205] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6328.051332] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6328.055578] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6328.077174] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6328.077187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6328.077190] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6328.077202] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6328.077206] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6328.077212] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6328.077214] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6328.077216] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6328.077219] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6328.077221] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6328.077224] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6328.077227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6328.077229] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6328.077237] Call Trace: [ 6328.077243] [ 6328.077248] ? show_regs.cold.9+0x22/0x2f [ 6328.077260] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6328.077267] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6328.077272] ? watchdog+0x40/0x40 [ 6328.077276] ? hrtimer_interrupt+0x128/0x2c0 [ 6328.077280] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6328.077287] ? apic_timer_interrupt+0xf/0x20 [ 6328.077291] [ 6328.077295] ? native_safe_halt+0x2e/0x50 [ 6328.077298] kvm_wait+0xcd/0xe0 [ 6328.077306] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6328.077314] ? dump_stack+0xf6/0x10e [ 6328.077320] do_raw_spin_lock+0x177/0x190 [ 6328.077326] _raw_spin_lock+0x21/0x30 [ 6328.077334] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6328.077633] obd_mod_free+0x46/0x140 [ptlrpc] [ 6328.077917] mdc_commit_open+0x91/0xb0 [mdc] [ 6328.077953] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6328.078163] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6328.078359] after_reply+0xe2c/0x1780 [ptlrpc] [ 6328.078566] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6328.078777] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6328.079225] ? do_wait_intr+0xf0/0xf0 [ 6328.079236] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6328.079452] kthread+0x1d1/0x200 [ 6328.079458] ? set_kthread_struct+0x70/0x70 [ 6328.079462] ret_from_fork+0x1f/0x30 [ 6328.182088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6328.199751] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6328.203713] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6328.207757] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6328.211772] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6328.218111] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6328.220485] RAX: 0000000000007e8d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6328.222777] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6328.225192] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6328.227999] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6328.230259] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6328.233178] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6328.235422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6328.237399] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6328.240123] Call Trace: [ 6328.240856] [ 6328.241372] ? show_regs.cold.9+0x22/0x2f [ 6328.243447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6328.245110] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6328.246333] ? watchdog+0x40/0x40 [ 6328.247251] ? hrtimer_interrupt+0x128/0x2c0 [ 6328.248590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6328.250199] ? apic_timer_interrupt+0xf/0x20 [ 6328.251619] [ 6328.254030] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6328.256362] do_raw_spin_lock+0x177/0x190 [ 6328.257582] _raw_spin_lock+0x21/0x30 [ 6328.258905] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6328.260953] ? do_raw_spin_unlock+0x75/0x190 [ 6328.262557] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6328.264534] ? do_wait_intr+0xf0/0xf0 [ 6328.266314] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6328.268091] mdc_close+0x53b/0xd70 [mdc] [ 6328.269376] lmv_close+0x18d/0x490 [lmv] [ 6328.270775] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6328.272467] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6328.274901] ll_md_close+0x274/0x800 [lustre] [ 6328.276589] ll_file_release+0x128/0xc50 [lustre] [ 6328.279431] __fput+0x10d/0x440 [ 6328.280441] ____fput+0x12/0x20 [ 6328.281433] task_work_run+0xc6/0x110 [ 6328.285496] exit_to_usermode_loop+0x1dd/0x1f0 [ 6328.288313] do_syscall_64+0x3d6/0x440 [ 6328.289626] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6328.291342] RIP: 0033:0x7fdc0afbfcd8 [ 6328.292978] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6328.301591] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6328.304409] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6328.306552] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6328.310239] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6328.314189] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6328.316343] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6330.847119] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4305s! [ 6330.851888] Showing busy workqueues and worker pools: [ 6330.855511] workqueue events: flags=0x0 [ 6330.856629] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6330.866907] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6330.880900] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6330.888799] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6330.899624] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6330.905348] pending: kfree_rcu_monitor [ 6330.906892] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6330.914132] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6330.928172] workqueue events_long: flags=0x0 [ 6330.929487] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6330.934103] pending: lu_objects_flush [obdclass] [ 6330.938586] workqueue events_unbound: flags=0x2 [ 6330.942965] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6330.948839] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6330.954000] workqueue events_power_efficient: flags=0x80 [ 6330.960818] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6330.963553] pending: neigh_periodic_work, check_lifetime [ 6330.966977] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6330.974963] pending: do_cache_clean, neigh_periodic_work [ 6330.980769] workqueue mm_percpu_wq: flags=0x8 [ 6330.983542] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6330.988060] pending: lru_add_drain_per_cpu [ 6330.992919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6330.996465] pending: vmstat_update, lru_add_drain_per_cpu [ 6330.999111] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6331.003562] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6331.011619] workqueue writeback: flags=0x4a [ 6331.012713] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6331.017915] pending: wb_workfn, wb_workfn [ 6331.020134] workqueue xprtiod: flags=0xa [ 6331.024916] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6331.028965] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6331.035857] workqueue ipv6_addrconf: flags=0x40008 [ 6331.042150] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6331.047807] pending: addrconf_verify_work [ 6331.050161] workqueue knbd0-recv: flags=0x1a [ 6331.058048] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6331.064152] in-flight: 655:recv_work [ 6331.067947] workqueue ptlrpc_pinger: flags=0x2 [ 6331.073193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6331.075040] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6331.081966] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4306s workers=4 idle: 36396 38 11188 [ 6331.088096] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6321s workers=2 idle: 658 [ 6332.051024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6332.058693] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6332.099706] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6332.109729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6332.119900] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6332.130762] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6332.136260] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6332.152305] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6332.156438] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6332.160023] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6332.164918] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6332.169380] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6332.171340] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6332.175819] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6332.180171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6332.183124] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6332.188432] Call Trace: [ 6332.189717] [ 6332.191090] ? show_regs.cold.9+0x22/0x2f [ 6332.193748] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6332.194922] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6332.199480] ? watchdog+0x40/0x40 [ 6332.200297] ? hrtimer_interrupt+0x128/0x2c0 [ 6332.201642] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6332.205539] ? apic_timer_interrupt+0xf/0x20 [ 6332.206787] [ 6332.207414] ? native_safe_halt+0x2e/0x50 [ 6332.208692] kvm_wait+0xcd/0xe0 [ 6332.210891] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6332.213751] do_raw_spin_lock+0x177/0x190 [ 6332.216734] _raw_spin_lock+0x21/0x30 [ 6332.218231] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6332.224423] process_one_work+0x2c8/0x700 [ 6332.229977] worker_thread+0x296/0x6e0 [ 6332.232119] ? rescuer_thread+0x570/0x570 [ 6332.235350] kthread+0x1d1/0x200 [ 6332.236586] ? set_kthread_struct+0x70/0x70 [ 6332.238271] ret_from_fork+0x1f/0x30 [ 6352.063032] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6352.077124] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6352.162460] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6352.188489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6352.200152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6352.214185] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6352.248147] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6352.260853] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6352.263308] RDX: 0000000000003414 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6352.283567] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6352.296184] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6352.306153] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6352.318022] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6352.336834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6352.340824] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6352.346143] Call Trace: [ 6352.349291] [ 6352.351874] ? show_regs.cold.9+0x22/0x2f [ 6352.354776] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6352.357344] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6352.363312] ? watchdog+0x40/0x40 [ 6352.366010] ? hrtimer_interrupt+0x128/0x2c0 [ 6352.371848] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6352.381318] ? apic_timer_interrupt+0xf/0x20 [ 6352.387187] [ 6352.392301] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6352.396208] do_raw_spin_lock+0x177/0x190 [ 6352.398286] _raw_spin_lock+0x21/0x30 [ 6352.400200] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6352.405094] ? do_raw_spin_unlock+0x75/0x190 [ 6352.411909] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6352.414585] ? do_wait_intr+0xf0/0xf0 [ 6352.418177] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6352.426957] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6352.437567] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6352.439829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6352.466175] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6352.478501] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6352.486167] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6352.490885] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6352.503991] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6352.515476] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6352.517147] lmv_intent_open+0x338/0x1150 [lmv] [ 6352.522826] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6352.524220] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6352.525611] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6352.530265] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6352.533537] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6352.536072] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6352.545598] ? do_raw_spin_unlock+0x75/0x190 [ 6352.553346] ? _raw_spin_unlock+0x12/0x30 [ 6352.555876] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6352.564457] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6352.568812] ll_atomic_open+0x36a/0xd70 [lustre] [ 6352.571815] lookup_open+0x356/0xb40 [ 6352.573466] do_last+0x313/0xfc0 [ 6352.575098] ? nd_jump_root+0xe5/0x160 [ 6352.577382] ? path_init+0x437/0x520 [ 6352.579194] path_openat+0xf7/0x500 [ 6352.580463] ? do_raw_spin_unlock+0x75/0x190 [ 6352.581931] do_filp_open+0x99/0x140 [ 6352.583128] ? getname_flags+0x6e/0x330 [ 6352.584512] ? __check_object_size+0xff/0x256 [ 6352.586313] ? do_raw_spin_unlock+0x75/0x190 [ 6352.588333] ? _raw_spin_unlock+0x12/0x30 [ 6352.589790] do_sys_openat2+0x2b4/0x410 [ 6352.591172] do_sys_open+0x73/0xa0 [ 6352.592452] __x64_sys_openat+0x24/0x30 [ 6352.594466] do_syscall_64+0xc1/0x440 [ 6352.595895] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6352.597616] RIP: 0033:0x7fdc0afbf332 [ 6352.598997] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6352.607418] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6352.614087] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6352.620033] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6352.628973] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6352.631374] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6352.637528] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6356.052449] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6356.054351] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6356.059286] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6356.059383] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6356.059393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6356.059406] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6356.059425] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6356.059430] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6356.059435] RAX: 0000000000007c10 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6356.059438] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6356.059441] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6356.059444] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6356.059446] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6356.059450] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6356.059453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6356.059456] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6356.059466] Call Trace: [ 6356.059473] [ 6356.059481] ? show_regs.cold.9+0x22/0x2f [ 6356.059494] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6356.059502] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6356.059507] ? watchdog+0x40/0x40 [ 6356.059512] ? hrtimer_interrupt+0x128/0x2c0 [ 6356.059517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6356.059532] ? apic_timer_interrupt+0xf/0x20 [ 6356.059536] [ 6356.059540] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6356.059546] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6356.059550] ? dump_stack+0xf6/0x10e [ 6356.059559] do_raw_spin_lock+0x177/0x190 [ 6356.059564] _raw_spin_lock+0x21/0x30 [ 6356.059581] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6356.109362] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6356.109453] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6356.109464] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6356.109468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6356.109487] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6356.109492] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6356.109496] RAX: 0000000000003b92 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6356.109499] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6356.109501] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6356.109503] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6356.109506] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6356.109510] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6356.109513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6356.109516] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6356.109525] Call Trace: [ 6356.109532] [ 6356.109540] ? show_regs.cold.9+0x22/0x2f [ 6356.109552] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6356.109560] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6356.109565] ? watchdog+0x40/0x40 [ 6356.109580] ? hrtimer_interrupt+0x128/0x2c0 [ 6356.109585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6356.109595] ? apic_timer_interrupt+0xf/0x20 [ 6356.109600] [ 6356.109603] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6356.109612] do_raw_spin_lock+0x177/0x190 [ 6356.109617] _raw_spin_lock+0x21/0x30 [ 6356.109622] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6356.109982] ? do_raw_spin_unlock+0x75/0x190 [ 6356.109988] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6356.110486] ? do_wait_intr+0xf0/0xf0 [ 6356.110497] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6356.110941] mdc_close+0x53b/0xd70 [mdc] [ 6356.111249] lmv_close+0x18d/0x490 [lmv] [ 6356.111285] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6356.111415] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6356.111495] ll_md_close+0x274/0x800 [lustre] [ 6356.111591] ll_file_release+0x128/0xc50 [lustre] [ 6356.111677] __fput+0x10d/0x440 [ 6356.111702] ____fput+0x12/0x20 [ 6356.111706] task_work_run+0xc6/0x110 [ 6356.111716] exit_to_usermode_loop+0x1dd/0x1f0 [ 6356.111725] do_syscall_64+0x3d6/0x440 [ 6356.111731] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6356.111742] RIP: 0033:0x7fdc0afbfcd8 [ 6356.111751] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6356.111755] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6356.111762] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6356.111764] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6356.111766] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6356.111769] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6356.111771] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6356.627861] obd_mod_free+0x46/0x140 [ptlrpc] [ 6356.647317] mdc_commit_open+0x91/0xb0 [mdc] [ 6356.648926] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6356.652260] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6356.664307] after_reply+0xe2c/0x1780 [ptlrpc] [ 6356.681683] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6356.689225] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6356.695900] ? do_wait_intr+0xf0/0xf0 [ 6356.702534] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6356.713000] kthread+0x1d1/0x200 [ 6356.714092] ? set_kthread_struct+0x70/0x70 [ 6356.724526] ret_from_fork+0x1f/0x30 [ 6360.051018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6360.054232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6360.118274] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6360.128223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6360.134579] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6360.142575] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6360.146240] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6360.171325] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6360.188873] RAX: 0000000000006084 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6360.199471] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6360.214416] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6360.217517] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6360.222959] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6360.227265] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6360.234096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6360.236604] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6360.241053] Call Trace: [ 6360.243243] [ 6360.244343] ? show_regs.cold.9+0x22/0x2f [ 6360.247361] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6360.249428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6360.255199] ? watchdog+0x40/0x40 [ 6360.256923] ? hrtimer_interrupt+0x128/0x2c0 [ 6360.265267] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6360.271369] ? apic_timer_interrupt+0xf/0x20 [ 6360.273578] [ 6360.287696] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6360.292882] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6360.298172] do_raw_spin_lock+0x177/0x190 [ 6360.304383] _raw_spin_lock+0x21/0x30 [ 6360.306929] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6360.315302] process_one_work+0x2c8/0x700 [ 6360.317484] worker_thread+0x296/0x6e0 [ 6360.322961] ? rescuer_thread+0x570/0x570 [ 6360.327384] kthread+0x1d1/0x200 [ 6360.329062] ? set_kthread_struct+0x70/0x70 [ 6360.331395] ret_from_fork+0x1f/0x30 [ 6361.567529] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4336s! [ 6361.580712] Showing busy workqueues and worker pools: [ 6361.583848] workqueue events: flags=0x0 [ 6361.587456] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6361.591763] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6361.596121] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6361.600053] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6361.606301] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6361.613853] pending: kfree_rcu_monitor [ 6361.617381] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6361.624419] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6361.648919] workqueue events_long: flags=0x0 [ 6361.654694] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6361.665753] pending: lu_objects_flush [obdclass] [ 6361.675555] workqueue events_unbound: flags=0x2 [ 6361.679754] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6361.682566] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6361.686928] workqueue events_power_efficient: flags=0x80 [ 6361.689237] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6361.691897] pending: neigh_periodic_work, check_lifetime [ 6361.697402] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6361.700017] pending: do_cache_clean, neigh_periodic_work [ 6361.702428] workqueue mm_percpu_wq: flags=0x8 [ 6361.704319] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6361.707145] pending: lru_add_drain_per_cpu [ 6361.708919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6361.712632] pending: vmstat_update, lru_add_drain_per_cpu [ 6361.715383] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6361.718579] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6361.725927] workqueue writeback: flags=0x4a [ 6361.728780] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6361.731694] pending: wb_workfn, wb_workfn [ 6361.733758] workqueue xprtiod: flags=0xa [ 6361.735638] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6361.739447] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6361.747310] workqueue ipv6_addrconf: flags=0x40008 [ 6361.749437] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6361.752389] pending: addrconf_verify_work [ 6361.754048] workqueue knbd0-recv: flags=0x1a [ 6361.755696] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6361.757955] in-flight: 655:recv_work [ 6361.764799] workqueue ptlrpc_pinger: flags=0x2 [ 6361.767526] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6361.774561] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6361.782041] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4336s workers=4 idle: 36396 38 11188 [ 6361.788749] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6352s workers=2 idle: 658 [ 6380.053199] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6380.056572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6380.099117] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6380.106069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6380.116073] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6380.119866] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6380.140228] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6380.144012] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6380.146245] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6380.155305] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6380.162538] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6380.168045] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6380.171438] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6380.175830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6380.180685] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6380.187680] Call Trace: [ 6380.189393] [ 6380.191305] ? show_regs.cold.9+0x22/0x2f [ 6380.194277] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6380.198244] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6380.199731] ? watchdog+0x40/0x40 [ 6380.207645] ? hrtimer_interrupt+0x128/0x2c0 [ 6380.210568] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6380.213953] ? apic_timer_interrupt+0xf/0x20 [ 6380.218408] [ 6380.220935] ? native_safe_halt+0x2e/0x50 [ 6380.224329] kvm_wait+0xcd/0xe0 [ 6380.225422] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6380.231886] do_raw_spin_lock+0x177/0x190 [ 6380.233979] _raw_spin_lock+0x21/0x30 [ 6380.245471] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6380.249526] ? do_raw_spin_unlock+0x75/0x190 [ 6380.255759] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6380.259644] ? do_wait_intr+0xf0/0xf0 [ 6380.267548] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6380.271547] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6380.276197] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6380.279927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6380.284907] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6380.289762] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6380.293731] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6380.297312] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6380.300857] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6380.304328] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6380.308929] lmv_intent_open+0x338/0x1150 [lmv] [ 6380.315578] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6380.319404] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6380.322634] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6380.324975] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6380.328551] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6380.334473] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6380.339529] ? do_raw_spin_unlock+0x75/0x190 [ 6380.342929] ? _raw_spin_unlock+0x12/0x30 [ 6380.344906] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6380.346904] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6380.349288] ll_atomic_open+0x36a/0xd70 [lustre] [ 6380.353749] lookup_open+0x356/0xb40 [ 6380.359590] do_last+0x313/0xfc0 [ 6380.362596] ? nd_jump_root+0xe5/0x160 [ 6380.364485] ? path_init+0x437/0x520 [ 6380.367382] path_openat+0xf7/0x500 [ 6380.368573] ? do_raw_spin_unlock+0x75/0x190 [ 6380.369598] do_filp_open+0x99/0x140 [ 6380.370576] ? getname_flags+0x6e/0x330 [ 6380.376321] ? __check_object_size+0xff/0x256 [ 6380.379211] ? do_raw_spin_unlock+0x75/0x190 [ 6380.382921] ? _raw_spin_unlock+0x12/0x30 [ 6380.386561] do_sys_openat2+0x2b4/0x410 [ 6380.390360] do_sys_open+0x73/0xa0 [ 6380.392893] __x64_sys_openat+0x24/0x30 [ 6380.396955] do_syscall_64+0xc1/0x440 [ 6380.400336] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6380.403333] RIP: 0033:0x7fdc0afbf332 [ 6380.404676] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6380.420640] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6380.426588] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6380.435559] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6380.440345] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6380.446215] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6380.449898] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6384.054238] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6384.056894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6384.101908] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6384.118001] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6384.123119] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6384.128024] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6384.141614] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6384.148615] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6384.165293] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6384.175491] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6384.186313] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6384.197465] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6384.207105] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6384.220755] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6384.231601] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6384.237865] Call Trace: [ 6384.244160] [ 6384.247953] ? show_regs.cold.9+0x22/0x2f [ 6384.252240] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6384.262393] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6384.266466] ? watchdog+0x40/0x40 [ 6384.274182] ? hrtimer_interrupt+0x128/0x2c0 [ 6384.287134] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6384.299231] ? apic_timer_interrupt+0xf/0x20 [ 6384.305598] [ 6384.306299] ? native_safe_halt+0x2e/0x50 [ 6384.316797] kvm_wait+0xcd/0xe0 [ 6384.318306] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6384.319900] do_raw_spin_lock+0x177/0x190 [ 6384.329062] _raw_spin_lock+0x21/0x30 [ 6384.338461] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6384.346032] ? do_raw_spin_unlock+0x75/0x190 [ 6384.348457] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6384.358561] ? do_wait_intr+0xf0/0xf0 [ 6384.361559] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6384.369037] mdc_close+0x53b/0xd70 [mdc] [ 6384.372514] lmv_close+0x18d/0x490 [lmv] [ 6384.378094] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6384.388487] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6384.399450] ll_md_close+0x274/0x800 [lustre] [ 6384.404704] ll_file_release+0x128/0xc50 [lustre] [ 6384.409887] __fput+0x10d/0x440 [ 6384.411766] ____fput+0x12/0x20 [ 6384.413341] task_work_run+0xc6/0x110 [ 6384.415745] exit_to_usermode_loop+0x1dd/0x1f0 [ 6384.417811] do_syscall_64+0x3d6/0x440 [ 6384.419742] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6384.422034] RIP: 0033:0x7fdc0afbfcd8 [ 6384.424054] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6384.436632] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6384.442319] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6384.447602] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6384.450779] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6384.456847] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6384.460500] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6388.048158] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 6388.050792] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6388.052136] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6388.067059] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6388.067077] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6388.067080] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6388.067099] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6388.067103] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6388.067108] RAX: 0000000000001d87 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6388.067110] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6388.067112] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6388.067114] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6388.067116] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6388.067119] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6388.067121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6388.067123] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6388.067130] Call Trace: [ 6388.067135] [ 6388.067140] ? show_regs.cold.9+0x22/0x2f [ 6388.067148] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6388.067153] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6388.067157] ? watchdog+0x40/0x40 [ 6388.067161] ? hrtimer_interrupt+0x128/0x2c0 [ 6388.067164] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6388.067171] ? apic_timer_interrupt+0xf/0x20 [ 6388.067175] [ 6388.067177] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6388.067181] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6388.067184] ? dump_stack+0xf6/0x10e [ 6388.067189] do_raw_spin_lock+0x177/0x190 [ 6388.067202] _raw_spin_lock+0x21/0x30 [ 6388.067209] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6388.067511] obd_mod_free+0x46/0x140 [ptlrpc] [ 6388.067730] mdc_commit_open+0x91/0xb0 [mdc] [ 6388.067772] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6388.067973] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6388.068172] after_reply+0xe2c/0x1780 [ptlrpc] [ 6388.068365] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6388.068558] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6388.069000] ? do_wait_intr+0xf0/0xf0 [ 6388.069009] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6388.069299] kthread+0x1d1/0x200 [ 6388.069305] ? set_kthread_struct+0x70/0x70 [ 6388.069310] ret_from_fork+0x1f/0x30 [ 6388.162198] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6388.179746] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6388.183516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6388.187340] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6388.190016] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6388.191574] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6388.197985] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6388.200713] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6388.203141] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6388.205908] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6388.208574] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6388.211978] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6388.214378] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6388.217389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6388.219120] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6388.221576] Call Trace: [ 6388.222538] [ 6388.223280] ? show_regs.cold.9+0x22/0x2f [ 6388.224735] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6388.226347] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6388.229212] ? watchdog+0x40/0x40 [ 6388.230404] ? hrtimer_interrupt+0x128/0x2c0 [ 6388.231845] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6388.233495] ? apic_timer_interrupt+0xf/0x20 [ 6388.235292] [ 6388.236235] ? native_safe_halt+0x2e/0x50 [ 6388.237619] kvm_wait+0xcd/0xe0 [ 6388.239017] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6388.241609] do_raw_spin_lock+0x177/0x190 [ 6388.243365] _raw_spin_lock+0x21/0x30 [ 6388.245051] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6388.247169] process_one_work+0x2c8/0x700 [ 6388.248613] worker_thread+0x296/0x6e0 [ 6388.250099] ? rescuer_thread+0x570/0x570 [ 6388.251518] kthread+0x1d1/0x200 [ 6388.252585] ? set_kthread_struct+0x70/0x70 [ 6388.254689] ret_from_fork+0x1f/0x30 [ 6392.287091] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4367s! [ 6392.289928] Showing busy workqueues and worker pools: [ 6392.291488] workqueue events: flags=0x0 [ 6392.296868] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6392.309465] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6392.319928] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6392.327875] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6392.347512] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6392.353113] pending: kfree_rcu_monitor [ 6392.361253] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6392.367405] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6392.378467] workqueue events_long: flags=0x0 [ 6392.383409] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6392.389724] pending: lu_objects_flush [obdclass] [ 6392.397205] workqueue events_unbound: flags=0x2 [ 6392.399401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6392.412153] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6392.427548] workqueue events_power_efficient: flags=0x80 [ 6392.437586] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6392.447788] pending: neigh_periodic_work, check_lifetime [ 6392.455314] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6392.462747] pending: do_cache_clean, neigh_periodic_work [ 6392.468171] workqueue mm_percpu_wq: flags=0x8 [ 6392.470847] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6392.477449] pending: lru_add_drain_per_cpu [ 6392.479960] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6392.483922] pending: vmstat_update, lru_add_drain_per_cpu [ 6392.487227] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6392.491028] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6392.494480] workqueue writeback: flags=0x4a [ 6392.496966] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6392.500523] pending: wb_workfn, wb_workfn [ 6392.502543] workqueue xprtiod: flags=0xa [ 6392.512433] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6392.515251] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6392.521288] workqueue ipv6_addrconf: flags=0x40008 [ 6392.526024] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6392.530972] pending: addrconf_verify_work [ 6392.533215] workqueue knbd0-recv: flags=0x1a [ 6392.537839] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6392.540308] in-flight: 655:recv_work [ 6392.545356] workqueue ptlrpc_pinger: flags=0x2 [ 6392.552910] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6392.556134] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6392.560347] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4367s workers=4 idle: 36396 38 11188 [ 6392.569464] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6383s workers=2 idle: 658 [ 6399.440126] rcu: INFO: rcu_sched self-detected stall on CPU [ 6399.444113] rcu: 0-...!: (2649890 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 6399.447658] (t=4380097 jiffies g=270761 q=3714) [ 6399.449712] rcu: rcu_sched kthread starved for 4375029 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6399.454316] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6399.457970] rcu: RCU grace-period kthread stack dump: [ 6399.460160] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6399.464247] Call Trace: [ 6399.465334] __schedule+0x351/0xcb0 [ 6399.466703] schedule+0xc0/0x180 [ 6399.468210] schedule_timeout+0xb4/0x190 [ 6399.469802] ? do_raw_spin_unlock+0x75/0x190 [ 6399.471590] ? __next_timer_interrupt+0x160/0x160 [ 6399.473537] ? prepare_to_swait_event+0xf3/0x140 [ 6399.475559] rcu_gp_kthread+0x8dd/0x15b0 [ 6399.476974] ? force_qs_rnp+0x2c0/0x2c0 [ 6399.478696] kthread+0x1d1/0x200 [ 6399.480117] ? set_kthread_struct+0x70/0x70 [ 6399.482061] ret_from_fork+0x1f/0x30 [ 6399.483640] rcu: Stack dump where RCU GP kthread last ran: [ 6399.486055] Sending NMI from CPU 0 to CPUs 1: [ 6399.488103] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 6399.488992] NMI backtrace for cpu 0 [ 6399.492767] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6399.497993] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6399.501906] Call Trace: [ 6399.502991] [ 6399.503940] ? dump_stack+0xbb/0x10e [ 6399.505598] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6399.507959] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6399.510005] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6399.512544] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6399.514921] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6399.517320] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6399.519583] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6399.522300] ? update_process_times+0x6d/0xc0 [ 6399.524281] ? tick_sched_handle.isra.18+0x27/0x90 [ 6399.526506] ? tick_sched_timer+0x53/0xb0 [ 6399.528551] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6399.530519] ? tick_sched_handle.isra.18+0x90/0x90 [ 6399.532861] ? hrtimer_interrupt+0x128/0x2c0 [ 6399.534821] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6399.536940] ? apic_timer_interrupt+0xf/0x20 [ 6399.539087] [ 6399.539916] ? native_safe_halt+0x2e/0x50 [ 6399.541704] ? kvm_wait+0xcd/0xe0 [ 6399.543445] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6399.546149] ? dump_stack+0xf6/0x10e [ 6399.547852] ? do_raw_spin_lock+0x177/0x190 [ 6399.549849] ? _raw_spin_lock+0x21/0x30 [ 6399.551871] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6399.554618] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6399.557433] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6399.560260] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6399.563606] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6399.566790] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6399.570160] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6399.573062] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6399.575280] ? do_wait_intr+0xf0/0xf0 [ 6399.576826] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6399.579342] ? kthread+0x1d1/0x200 [ 6399.580319] ? set_kthread_struct+0x70/0x70 [ 6399.581985] ? ret_from_fork+0x1f/0x30 [ 6408.056881] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6408.060484] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6408.093816] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6408.105627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6408.109934] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6408.113510] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6408.128488] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6408.136711] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6408.145724] RDX: 00000000000013a3 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6408.157578] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6408.170828] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6408.175553] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6408.183999] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6408.201968] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6408.209783] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6408.212533] Call Trace: [ 6408.213706] [ 6408.214868] ? show_regs.cold.9+0x22/0x2f [ 6408.216145] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6408.221472] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6408.227020] ? watchdog+0x40/0x40 [ 6408.229449] ? hrtimer_interrupt+0x128/0x2c0 [ 6408.231353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6408.233963] ? apic_timer_interrupt+0xf/0x20 [ 6408.239242] [ 6408.243021] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6408.245579] do_raw_spin_lock+0x177/0x190 [ 6408.247869] _raw_spin_lock+0x21/0x30 [ 6408.251898] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6408.254778] ? do_raw_spin_unlock+0x75/0x190 [ 6408.258896] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6408.268788] ? do_wait_intr+0xf0/0xf0 [ 6408.271028] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6408.278863] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6408.282540] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6408.287158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6408.293375] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6408.294858] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6408.296604] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6408.298145] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6408.299928] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6408.305881] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6408.311551] lmv_intent_open+0x338/0x1150 [lmv] [ 6408.317883] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6408.324139] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6408.326858] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6408.328752] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6408.330536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6408.335031] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6408.337857] ? do_raw_spin_unlock+0x75/0x190 [ 6408.339045] ? _raw_spin_unlock+0x12/0x30 [ 6408.341866] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6408.344038] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6408.346241] ll_atomic_open+0x36a/0xd70 [lustre] [ 6408.348823] lookup_open+0x356/0xb40 [ 6408.349809] do_last+0x313/0xfc0 [ 6408.350957] ? nd_jump_root+0xe5/0x160 [ 6408.352185] ? path_init+0x437/0x520 [ 6408.354690] path_openat+0xf7/0x500 [ 6408.355817] ? do_raw_spin_unlock+0x75/0x190 [ 6408.359023] do_filp_open+0x99/0x140 [ 6408.360997] ? getname_flags+0x6e/0x330 [ 6408.363118] ? __check_object_size+0xff/0x256 [ 6408.365155] ? do_raw_spin_unlock+0x75/0x190 [ 6408.370986] ? _raw_spin_unlock+0x12/0x30 [ 6408.372009] do_sys_openat2+0x2b4/0x410 [ 6408.373060] do_sys_open+0x73/0xa0 [ 6408.374156] __x64_sys_openat+0x24/0x30 [ 6408.377443] do_syscall_64+0xc1/0x440 [ 6408.379574] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6408.381292] RIP: 0033:0x7fdc0afbf332 [ 6408.382665] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6408.391386] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6408.397600] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6408.403149] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6408.406448] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6408.409817] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6408.414602] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6412.056183] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6412.058900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6412.088401] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6412.098608] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6412.106702] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6412.109499] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6412.118643] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6412.121438] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6412.124363] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6412.127521] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6412.130574] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6412.133530] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6412.136327] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6412.139768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6412.142010] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6412.148366] Call Trace: [ 6412.149003] [ 6412.149710] ? show_regs.cold.9+0x22/0x2f [ 6412.151065] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6412.152770] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6412.154475] ? watchdog+0x40/0x40 [ 6412.155585] ? hrtimer_interrupt+0x128/0x2c0 [ 6412.157128] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6412.159169] ? apic_timer_interrupt+0xf/0x20 [ 6412.161145] [ 6412.161847] ? native_safe_halt+0x2e/0x50 [ 6412.163247] kvm_wait+0xcd/0xe0 [ 6412.166883] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6412.168880] do_raw_spin_lock+0x177/0x190 [ 6412.170863] _raw_spin_lock+0x21/0x30 [ 6412.174615] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6412.182657] ? do_raw_spin_unlock+0x75/0x190 [ 6412.190074] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6412.195671] ? do_wait_intr+0xf0/0xf0 [ 6412.201396] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6412.208786] mdc_close+0x53b/0xd70 [mdc] [ 6412.210699] lmv_close+0x18d/0x490 [lmv] [ 6412.212715] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6412.214832] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6412.223228] ll_md_close+0x274/0x800 [lustre] [ 6412.231554] ll_file_release+0x128/0xc50 [lustre] [ 6412.236527] __fput+0x10d/0x440 [ 6412.238289] ____fput+0x12/0x20 [ 6412.239414] task_work_run+0xc6/0x110 [ 6412.246338] exit_to_usermode_loop+0x1dd/0x1f0 [ 6412.249711] do_syscall_64+0x3d6/0x440 [ 6412.252036] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6412.256387] RIP: 0033:0x7fdc0afbfcd8 [ 6412.258298] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6412.268453] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6412.273422] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6412.279465] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6412.284850] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6412.295321] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6412.301052] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6423.008617] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4398s! [ 6423.015675] Showing busy workqueues and worker pools: [ 6423.018173] workqueue events: flags=0x0 [ 6423.020217] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6423.024373] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6423.032893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6423.037724] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6423.046301] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6423.054670] pending: kfree_rcu_monitor [ 6423.057285] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6423.064788] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6423.076490] workqueue events_long: flags=0x0 [ 6423.080870] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6423.087134] pending: lu_objects_flush [obdclass] [ 6423.090179] workqueue events_unbound: flags=0x2 [ 6423.092910] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6423.096118] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6423.102215] workqueue events_power_efficient: flags=0x80 [ 6423.105128] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6423.110110] pending: neigh_periodic_work, check_lifetime [ 6423.112901] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6423.116373] pending: do_cache_clean, neigh_periodic_work [ 6423.119060] workqueue mm_percpu_wq: flags=0x8 [ 6423.121352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6423.124416] pending: lru_add_drain_per_cpu [ 6423.129342] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6423.133338] pending: vmstat_update, lru_add_drain_per_cpu [ 6423.136160] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6423.140225] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6423.143156] workqueue writeback: flags=0x4a [ 6423.144985] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6423.149919] pending: wb_workfn, wb_workfn [ 6423.153170] workqueue xprtiod: flags=0xa [ 6423.155444] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6423.159367] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6423.164452] workqueue ipv6_addrconf: flags=0x40008 [ 6423.172362] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6423.175497] pending: addrconf_verify_work [ 6423.178065] workqueue knbd0-recv: flags=0x1a [ 6423.181801] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6423.185626] in-flight: 655:recv_work [ 6423.188944] workqueue ptlrpc_pinger: flags=0x2 [ 6423.191306] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6423.194541] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6423.198408] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4398s workers=4 idle: 36396 38 11188 [ 6423.203177] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6413s workers=2 idle: 658 [ 6424.051019] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6424.053240] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6424.072249] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6424.076651] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6424.080043] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6424.083105] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6424.085706] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6424.101204] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6424.109423] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6424.117337] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6424.132315] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6424.144677] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6424.150145] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6424.153650] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6424.157269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6424.159883] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6424.163123] Call Trace: [ 6424.164456] [ 6424.165393] ? show_regs.cold.9+0x22/0x2f [ 6424.167154] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6424.172595] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6424.175849] ? watchdog+0x40/0x40 [ 6424.181167] ? hrtimer_interrupt+0x128/0x2c0 [ 6424.186903] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6424.190345] ? apic_timer_interrupt+0xf/0x20 [ 6424.195621] [ 6424.197839] ? native_safe_halt+0x2e/0x50 [ 6424.204886] kvm_wait+0xcd/0xe0 [ 6424.212152] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6424.221062] do_raw_spin_lock+0x177/0x190 [ 6424.225244] _raw_spin_lock+0x21/0x30 [ 6424.228395] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6424.234606] process_one_work+0x2c8/0x700 [ 6424.238886] worker_thread+0x296/0x6e0 [ 6424.241904] ? rescuer_thread+0x570/0x570 [ 6424.245250] kthread+0x1d1/0x200 [ 6424.246569] ? set_kthread_struct+0x70/0x70 [ 6424.253588] ret_from_fork+0x1f/0x30 [ 6436.052100] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 6436.059386] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6436.117286] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6436.128495] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6436.138602] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6436.141460] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6436.160587] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6436.168781] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6436.172249] RDX: 0000000000002edf RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6436.176125] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6436.180126] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6436.183821] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6436.187989] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6436.191901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6436.194352] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6436.197418] Call Trace: [ 6436.198486] [ 6436.199375] ? show_regs.cold.9+0x22/0x2f [ 6436.201364] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6436.203199] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6436.204815] ? watchdog+0x40/0x40 [ 6436.206421] ? hrtimer_interrupt+0x128/0x2c0 [ 6436.208251] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6436.210711] ? apic_timer_interrupt+0xf/0x20 [ 6436.212505] [ 6436.213313] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6436.215561] do_raw_spin_lock+0x177/0x190 [ 6436.217532] _raw_spin_lock+0x21/0x30 [ 6436.219243] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6436.221674] ? do_raw_spin_unlock+0x75/0x190 [ 6436.223424] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6436.225627] ? do_wait_intr+0xf0/0xf0 [ 6436.227351] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6436.229985] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6436.232478] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6436.235569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6436.240862] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6436.243360] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6436.245223] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6436.247421] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6436.250212] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6436.252962] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6436.254734] lmv_intent_open+0x338/0x1150 [lmv] [ 6436.257111] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6436.259690] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6436.262349] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6436.263901] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6436.265521] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6436.267984] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6436.270346] ? do_raw_spin_unlock+0x75/0x190 [ 6436.272355] ? _raw_spin_unlock+0x12/0x30 [ 6436.273694] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6436.276185] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6436.278618] ll_atomic_open+0x36a/0xd70 [lustre] [ 6436.281735] lookup_open+0x356/0xb40 [ 6436.283428] do_last+0x313/0xfc0 [ 6436.285427] ? nd_jump_root+0xe5/0x160 [ 6436.287658] ? path_init+0x437/0x520 [ 6436.290180] path_openat+0xf7/0x500 [ 6436.291972] ? do_raw_spin_unlock+0x75/0x190 [ 6436.294386] do_filp_open+0x99/0x140 [ 6436.296955] ? getname_flags+0x6e/0x330 [ 6436.299396] ? __check_object_size+0xff/0x256 [ 6436.301624] ? do_raw_spin_unlock+0x75/0x190 [ 6436.303964] ? _raw_spin_unlock+0x12/0x30 [ 6436.306268] do_sys_openat2+0x2b4/0x410 [ 6436.308592] do_sys_open+0x73/0xa0 [ 6436.310475] __x64_sys_openat+0x24/0x30 [ 6436.312710] do_syscall_64+0xc1/0x440 [ 6436.315077] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6436.317374] RIP: 0033:0x7fdc0afbf332 [ 6436.319681] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6436.329946] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6436.334723] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6436.339158] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6436.343313] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6436.347741] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6436.352424] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6440.060166] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6440.078862] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6440.136763] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6440.151040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6440.162367] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6440.173161] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6440.190702] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6440.196231] RAX: 0000000000005586 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6440.202305] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6440.206532] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6440.211293] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6440.221274] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6440.226791] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6440.235777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6440.237907] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6440.242572] Call Trace: [ 6440.244226] [ 6440.245289] ? show_regs.cold.9+0x22/0x2f [ 6440.248140] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6440.250615] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6440.253728] ? watchdog+0x40/0x40 [ 6440.255370] ? hrtimer_interrupt+0x128/0x2c0 [ 6440.259731] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6440.262324] ? apic_timer_interrupt+0xf/0x20 [ 6440.264694] [ 6440.266016] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6440.277015] do_raw_spin_lock+0x177/0x190 [ 6440.288787] _raw_spin_lock+0x21/0x30 [ 6440.294307] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6440.303524] ? do_raw_spin_unlock+0x75/0x190 [ 6440.322070] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6440.323959] ? do_wait_intr+0xf0/0xf0 [ 6440.328760] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6440.330787] mdc_close+0x53b/0xd70 [mdc] [ 6440.348863] lmv_close+0x18d/0x490 [lmv] [ 6440.359770] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6440.361896] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6440.378321] ll_md_close+0x274/0x800 [lustre] [ 6440.388432] ll_file_release+0x128/0xc50 [lustre] [ 6440.394369] __fput+0x10d/0x440 [ 6440.397296] ____fput+0x12/0x20 [ 6440.404733] task_work_run+0xc6/0x110 [ 6440.408191] exit_to_usermode_loop+0x1dd/0x1f0 [ 6440.421153] do_syscall_64+0x3d6/0x440 [ 6440.431594] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6440.438347] RIP: 0033:0x7fdc0afbfcd8 [ 6440.439651] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6440.453182] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6440.458417] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6440.467347] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6440.474596] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6440.483888] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6440.490881] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6448.053842] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 6448.062714] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6448.103944] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6448.118220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6448.137210] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6448.151070] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6448.192429] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6448.200945] RAX: 0000000000007d8d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6448.217033] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6448.222255] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6448.244195] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6448.261594] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6448.269553] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6448.274694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6448.277922] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6448.281516] Call Trace: [ 6448.282775] [ 6448.283759] ? show_regs.cold.9+0x22/0x2f [ 6448.285699] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6448.287799] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6448.291025] ? watchdog+0x40/0x40 [ 6448.292868] ? hrtimer_interrupt+0x128/0x2c0 [ 6448.294950] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6448.299530] ? apic_timer_interrupt+0xf/0x20 [ 6448.301783] [ 6448.302811] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6448.305658] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6448.308047] ? dump_stack+0xf6/0x10e [ 6448.310784] do_raw_spin_lock+0x177/0x190 [ 6448.313032] _raw_spin_lock+0x21/0x30 [ 6448.318288] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6448.333081] obd_mod_free+0x46/0x140 [ptlrpc] [ 6448.340731] mdc_commit_open+0x91/0xb0 [mdc] [ 6448.345111] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6448.348863] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6448.350808] after_reply+0xe2c/0x1780 [ptlrpc] [ 6448.352743] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6448.354688] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6448.356263] ? do_wait_intr+0xf0/0xf0 [ 6448.357360] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6448.359033] kthread+0x1d1/0x200 [ 6448.359881] ? set_kthread_struct+0x70/0x70 [ 6448.361199] ret_from_fork+0x1f/0x30 [ 6452.055133] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6452.066416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6452.127016] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6452.139123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6452.163098] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6452.179132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6452.187017] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6452.206510] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6452.217952] RAX: 0000000000007291 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6452.227956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6452.236334] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6452.244977] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6452.259746] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6452.267263] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6452.276799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6452.282359] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6452.296314] Call Trace: [ 6452.297220] [ 6452.298832] ? show_regs.cold.9+0x22/0x2f [ 6452.303914] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6452.315914] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6452.320164] ? watchdog+0x40/0x40 [ 6452.322316] ? hrtimer_interrupt+0x128/0x2c0 [ 6452.325919] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6452.331451] ? apic_timer_interrupt+0xf/0x20 [ 6452.337870] [ 6452.339089] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6452.343152] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6452.348266] do_raw_spin_lock+0x177/0x190 [ 6452.349603] _raw_spin_lock+0x21/0x30 [ 6452.355036] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6452.359670] process_one_work+0x2c8/0x700 [ 6452.360990] worker_thread+0x296/0x6e0 [ 6452.362249] ? rescuer_thread+0x570/0x570 [ 6452.369386] kthread+0x1d1/0x200 [ 6452.372127] ? set_kthread_struct+0x70/0x70 [ 6452.376000] ret_from_fork+0x1f/0x30 [ 6453.215618] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4428s! [ 6453.219209] Showing busy workqueues and worker pools: [ 6453.223422] workqueue events: flags=0x0 [ 6453.224676] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6453.231429] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6453.239057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6453.243044] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6453.250813] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6453.255760] pending: kfree_rcu_monitor [ 6453.258584] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6453.263740] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6453.276941] workqueue events_long: flags=0x0 [ 6453.282491] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6453.284488] pending: lu_objects_flush [obdclass] [ 6453.290218] workqueue events_unbound: flags=0x2 [ 6453.292855] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6453.297491] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6453.307238] workqueue events_power_efficient: flags=0x80 [ 6453.311787] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6453.316538] pending: neigh_periodic_work, check_lifetime [ 6453.324221] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6453.328169] pending: do_cache_clean, neigh_periodic_work [ 6453.331242] workqueue mm_percpu_wq: flags=0x8 [ 6453.333976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6453.339165] pending: lru_add_drain_per_cpu [ 6453.342987] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6453.353188] pending: vmstat_update, lru_add_drain_per_cpu [ 6453.354950] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6453.361936] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6453.366171] workqueue writeback: flags=0x4a [ 6453.368825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6453.372074] pending: wb_workfn, wb_workfn [ 6453.375907] workqueue xprtiod: flags=0xa [ 6453.380975] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6453.385258] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6453.389043] workqueue ipv6_addrconf: flags=0x40008 [ 6453.391300] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6453.397887] pending: addrconf_verify_work [ 6453.399410] workqueue knbd0-recv: flags=0x1a [ 6453.402091] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6453.405231] in-flight: 655:recv_work [ 6453.407112] workqueue ptlrpc_pinger: flags=0x2 [ 6453.409023] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6453.414458] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6453.417371] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4428s workers=4 idle: 36396 38 11188 [ 6453.420915] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6443s workers=2 idle: 658 [ 6464.053175] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 6464.063928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6464.095520] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6464.103932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6464.108758] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6464.112528] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6464.141404] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6464.149639] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6464.158261] RDX: 00000000000008c2 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6464.168066] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6464.173274] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6464.177595] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6464.186661] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6464.190462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6464.193806] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6464.198007] Call Trace: [ 6464.199251] [ 6464.201240] ? show_regs.cold.9+0x22/0x2f [ 6464.204012] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6464.206697] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6464.209016] ? watchdog+0x40/0x40 [ 6464.211084] ? hrtimer_interrupt+0x128/0x2c0 [ 6464.213587] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6464.216516] ? apic_timer_interrupt+0xf/0x20 [ 6464.218448] [ 6464.219423] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6464.221977] do_raw_spin_lock+0x177/0x190 [ 6464.223734] _raw_spin_lock+0x21/0x30 [ 6464.226456] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6464.234386] ? do_raw_spin_unlock+0x75/0x190 [ 6464.238047] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6464.243993] ? do_wait_intr+0xf0/0xf0 [ 6464.247622] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6464.257037] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6464.263644] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6464.270032] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6464.273958] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6464.278875] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6464.280121] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6464.281274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6464.287929] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6464.296713] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6464.300262] lmv_intent_open+0x338/0x1150 [lmv] [ 6464.306231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6464.310154] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6464.312696] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6464.322212] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6464.328986] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6464.336129] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6464.343518] ? do_raw_spin_unlock+0x75/0x190 [ 6464.350430] ? _raw_spin_unlock+0x12/0x30 [ 6464.356509] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6464.359621] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6464.363192] ll_atomic_open+0x36a/0xd70 [lustre] [ 6464.366227] lookup_open+0x356/0xb40 [ 6464.375154] do_last+0x313/0xfc0 [ 6464.380824] ? nd_jump_root+0xe5/0x160 [ 6464.389153] ? path_init+0x437/0x520 [ 6464.398153] path_openat+0xf7/0x500 [ 6464.403747] ? do_raw_spin_unlock+0x75/0x190 [ 6464.408541] do_filp_open+0x99/0x140 [ 6464.413142] ? getname_flags+0x6e/0x330 [ 6464.421761] ? __check_object_size+0xff/0x256 [ 6464.427221] ? do_raw_spin_unlock+0x75/0x190 [ 6464.432665] ? _raw_spin_unlock+0x12/0x30 [ 6464.438412] do_sys_openat2+0x2b4/0x410 [ 6464.443987] do_sys_open+0x73/0xa0 [ 6464.446128] __x64_sys_openat+0x24/0x30 [ 6464.448876] do_syscall_64+0xc1/0x440 [ 6464.454110] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6464.457824] RIP: 0033:0x7fdc0afbf332 [ 6464.461062] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6464.483260] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6464.492503] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6464.499957] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6464.507665] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6464.516098] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6464.525151] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6468.054026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6468.056645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6468.076345] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6468.081101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6468.084508] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6468.086450] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6468.094262] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6468.096264] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6468.097905] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6468.099626] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6468.103075] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6468.106213] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6468.109414] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6468.114052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6468.116084] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6468.119266] Call Trace: [ 6468.120311] [ 6468.121096] ? show_regs.cold.9+0x22/0x2f [ 6468.122573] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6468.124815] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6468.126889] ? watchdog+0x40/0x40 [ 6468.128348] ? hrtimer_interrupt+0x128/0x2c0 [ 6468.130126] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6468.132017] ? apic_timer_interrupt+0xf/0x20 [ 6468.133934] [ 6468.134957] ? native_safe_halt+0x2e/0x50 [ 6468.137429] kvm_wait+0xcd/0xe0 [ 6468.138701] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6468.140947] do_raw_spin_lock+0x177/0x190 [ 6468.142748] _raw_spin_lock+0x21/0x30 [ 6468.144045] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6468.146646] ? do_raw_spin_unlock+0x75/0x190 [ 6468.147793] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6468.150526] ? do_wait_intr+0xf0/0xf0 [ 6468.152102] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6468.154628] mdc_close+0x53b/0xd70 [mdc] [ 6468.156378] lmv_close+0x18d/0x490 [lmv] [ 6468.158192] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6468.160310] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6468.162982] ll_md_close+0x274/0x800 [lustre] [ 6468.165108] ll_file_release+0x128/0xc50 [lustre] [ 6468.167547] __fput+0x10d/0x440 [ 6468.169414] ____fput+0x12/0x20 [ 6468.170774] task_work_run+0xc6/0x110 [ 6468.172465] exit_to_usermode_loop+0x1dd/0x1f0 [ 6468.174477] do_syscall_64+0x3d6/0x440 [ 6468.176351] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6468.178317] RIP: 0033:0x7fdc0afbfcd8 [ 6468.180171] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6468.190721] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6468.194644] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6468.197645] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6468.201418] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6468.204467] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6468.207166] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6480.051652] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6480.055424] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6480.055628] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 6480.088109] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6480.088124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6480.088132] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6480.091648] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6480.091747] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6480.091755] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6480.091757] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6480.091774] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6480.091782] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6480.091789] RAX: 0000000000007085 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6480.091791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6480.091794] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6480.091796] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6480.091799] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6480.091802] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6480.091805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6480.091808] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6480.091824] Call Trace: [ 6480.091833] [ 6480.091839] ? show_regs.cold.9+0x22/0x2f [ 6480.091849] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6480.091860] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6480.091869] ? watchdog+0x40/0x40 [ 6480.091874] ? hrtimer_interrupt+0x128/0x2c0 [ 6480.091879] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6480.091888] ? apic_timer_interrupt+0xf/0x20 [ 6480.091896] [ 6480.091904] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6480.091909] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6480.091913] ? dump_stack+0xf6/0x10e [ 6480.091921] do_raw_spin_lock+0x177/0x190 [ 6480.091929] _raw_spin_lock+0x21/0x30 [ 6480.091935] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6480.092348] obd_mod_free+0x46/0x140 [ptlrpc] [ 6480.092873] mdc_commit_open+0x91/0xb0 [mdc] [ 6480.092903] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6480.093138] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6480.093378] after_reply+0xe2c/0x1780 [ptlrpc] [ 6480.093612] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6480.093813] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6480.100678] [ 6480.100687] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6480.100706] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6480.100711] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6480.100715] RAX: 0000000000006c17 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6480.100718] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6480.100720] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6480.100723] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6480.100726] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6480.100739] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6480.100742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6480.100744] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6480.100751] Call Trace: [ 6480.100757] [ 6480.100763] ? show_regs.cold.9+0x22/0x2f [ 6480.100772] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6480.100778] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6480.100782] ? watchdog+0x40/0x40 [ 6480.100786] ? hrtimer_interrupt+0x128/0x2c0 [ 6480.100790] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6480.100797] ? apic_timer_interrupt+0xf/0x20 [ 6480.100802] [ 6480.100805] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6480.100810] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6480.100818] do_raw_spin_lock+0x177/0x190 [ 6480.100823] _raw_spin_lock+0x21/0x30 [ 6480.100829] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6480.105741] ? do_wait_intr+0xf0/0xf0 [ 6480.105761] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6480.106026] kthread+0x1d1/0x200 [ 6480.106039] ? set_kthread_struct+0x70/0x70 [ 6480.106043] ret_from_fork+0x1f/0x30 [ 6480.509307] process_one_work+0x2c8/0x700 [ 6480.510923] worker_thread+0x296/0x6e0 [ 6480.515739] ? rescuer_thread+0x570/0x570 [ 6480.518052] kthread+0x1d1/0x200 [ 6480.520079] ? set_kthread_struct+0x70/0x70 [ 6480.542700] ret_from_fork+0x1f/0x30 [ 6483.935073] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4459s! [ 6483.938770] Showing busy workqueues and worker pools: [ 6483.941139] workqueue events: flags=0x0 [ 6483.942965] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6483.946136] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6483.949135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6483.953415] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6483.959156] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6483.962314] pending: kfree_rcu_monitor [ 6483.964091] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6483.967631] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6483.974184] workqueue events_long: flags=0x0 [ 6483.976470] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6483.979885] pending: lu_objects_flush [obdclass] [ 6483.982507] workqueue events_unbound: flags=0x2 [ 6483.984559] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6483.987810] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6483.994121] workqueue events_power_efficient: flags=0x80 [ 6483.996907] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6484.000292] pending: neigh_periodic_work, check_lifetime [ 6484.003275] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6484.006452] pending: do_cache_clean, neigh_periodic_work [ 6484.009400] workqueue mm_percpu_wq: flags=0x8 [ 6484.011466] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6484.014570] pending: lru_add_drain_per_cpu [ 6484.016799] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6484.021073] pending: vmstat_update, lru_add_drain_per_cpu [ 6484.024307] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6484.028030] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6484.031801] workqueue writeback: flags=0x4a [ 6484.033919] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6484.038011] pending: wb_workfn, wb_workfn [ 6484.040173] workqueue xprtiod: flags=0xa [ 6484.042433] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6484.045649] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6484.049963] workqueue ipv6_addrconf: flags=0x40008 [ 6484.052910] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6484.056884] pending: addrconf_verify_work [ 6484.059768] workqueue knbd0-recv: flags=0x1a [ 6484.061655] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6484.064315] in-flight: 655:recv_work [ 6484.065859] workqueue ptlrpc_pinger: flags=0x2 [ 6484.069407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6484.071162] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6484.074260] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4459s workers=4 idle: 36396 38 11188 [ 6484.077785] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6474s workers=2 idle: 658 [ 6492.053240] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 6492.066689] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6492.125670] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6492.156808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6492.159608] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6492.162915] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6492.190036] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6492.192736] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6492.204793] RDX: 00000000000076e3 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6492.211803] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6492.213747] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6492.223303] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6492.229007] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6492.235466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6492.242933] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6492.246900] Call Trace: [ 6492.247775] [ 6492.250796] ? show_regs.cold.9+0x22/0x2f [ 6492.252114] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6492.253727] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6492.259898] ? watchdog+0x40/0x40 [ 6492.261896] ? hrtimer_interrupt+0x128/0x2c0 [ 6492.263904] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6492.268199] ? apic_timer_interrupt+0xf/0x20 [ 6492.271620] [ 6492.272458] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6492.281914] do_raw_spin_lock+0x177/0x190 [ 6492.288575] _raw_spin_lock+0x21/0x30 [ 6492.294272] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6492.306214] ? do_raw_spin_unlock+0x75/0x190 [ 6492.312397] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6492.323790] ? do_wait_intr+0xf0/0xf0 [ 6492.324604] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6492.326035] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6492.338332] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6492.354626] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6492.367325] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6492.376889] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6492.378522] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6492.387844] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6492.392923] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6492.401941] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6492.411297] lmv_intent_open+0x338/0x1150 [lmv] [ 6492.414609] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6492.416476] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6492.427943] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6492.431842] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6492.435720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6492.444915] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6492.455395] ? do_raw_spin_unlock+0x75/0x190 [ 6492.461156] ? _raw_spin_unlock+0x12/0x30 [ 6492.462785] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6492.465474] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6492.469979] ll_atomic_open+0x36a/0xd70 [lustre] [ 6492.475471] lookup_open+0x356/0xb40 [ 6492.481991] do_last+0x313/0xfc0 [ 6492.483706] ? nd_jump_root+0xe5/0x160 [ 6492.489437] ? path_init+0x437/0x520 [ 6492.498372] path_openat+0xf7/0x500 [ 6492.506277] ? do_raw_spin_unlock+0x75/0x190 [ 6492.512673] do_filp_open+0x99/0x140 [ 6492.515875] ? getname_flags+0x6e/0x330 [ 6492.519262] ? __check_object_size+0xff/0x256 [ 6492.523369] ? do_raw_spin_unlock+0x75/0x190 [ 6492.527203] ? _raw_spin_unlock+0x12/0x30 [ 6492.530364] do_sys_openat2+0x2b4/0x410 [ 6492.533353] do_sys_open+0x73/0xa0 [ 6492.537887] __x64_sys_openat+0x24/0x30 [ 6492.541924] do_syscall_64+0xc1/0x440 [ 6492.546576] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6492.552745] RIP: 0033:0x7fdc0afbf332 [ 6492.556277] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6492.577353] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6492.590229] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6492.600804] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6492.603008] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6492.619053] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6492.637154] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6496.054028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6496.056653] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6496.074704] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6496.078684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6496.082317] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6496.084396] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6496.100222] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6496.103476] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6496.106462] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6496.109337] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6496.111864] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6496.114477] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6496.117657] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6496.121082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6496.123631] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6496.126448] Call Trace: [ 6496.127326] [ 6496.127966] ? show_regs.cold.9+0x22/0x2f [ 6496.129407] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6496.131284] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6496.132805] ? watchdog+0x40/0x40 [ 6496.134401] ? hrtimer_interrupt+0x128/0x2c0 [ 6496.136161] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6496.138333] ? apic_timer_interrupt+0xf/0x20 [ 6496.141143] [ 6496.142021] ? native_safe_halt+0x2e/0x50 [ 6496.143648] kvm_wait+0xcd/0xe0 [ 6496.145309] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6496.147466] do_raw_spin_lock+0x177/0x190 [ 6496.148891] _raw_spin_lock+0x21/0x30 [ 6496.150601] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6496.153539] ? do_raw_spin_unlock+0x75/0x190 [ 6496.155641] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6496.159320] ? do_wait_intr+0xf0/0xf0 [ 6496.160428] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6496.165190] mdc_close+0x53b/0xd70 [mdc] [ 6496.166407] lmv_close+0x18d/0x490 [lmv] [ 6496.168997] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6496.171276] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6496.173863] ll_md_close+0x274/0x800 [lustre] [ 6496.176079] ll_file_release+0x128/0xc50 [lustre] [ 6496.179017] __fput+0x10d/0x440 [ 6496.180116] ____fput+0x12/0x20 [ 6496.181512] task_work_run+0xc6/0x110 [ 6496.182952] exit_to_usermode_loop+0x1dd/0x1f0 [ 6496.184542] do_syscall_64+0x3d6/0x440 [ 6496.185922] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6496.187973] RIP: 0033:0x7fdc0afbfcd8 [ 6496.188986] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6496.194390] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6496.196496] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6496.198836] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6496.201128] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6496.203196] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6496.206514] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6508.055223] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6508.065476] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6508.097241] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6508.102762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6508.107436] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6508.111192] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6508.114380] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6508.128892] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6508.133308] RAX: 0000000000007d0a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6508.137028] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6508.140633] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6508.146130] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6508.150430] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6508.154695] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6508.161106] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6508.166340] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6508.173092] Call Trace: [ 6508.178278] [ 6508.179217] ? show_regs.cold.9+0x22/0x2f [ 6508.181902] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6508.183717] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6508.185669] ? watchdog+0x40/0x40 [ 6508.186807] ? hrtimer_interrupt+0x128/0x2c0 [ 6508.191890] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6508.197813] ? apic_timer_interrupt+0xf/0x20 [ 6508.199338] [ 6508.200344] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6508.203048] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6508.205276] do_raw_spin_lock+0x177/0x190 [ 6508.207044] _raw_spin_lock+0x21/0x30 [ 6508.212314] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6508.215492] process_one_work+0x2c8/0x700 [ 6508.216842] worker_thread+0x296/0x6e0 [ 6508.218777] ? rescuer_thread+0x570/0x570 [ 6508.220402] kthread+0x1d1/0x200 [ 6508.222084] ? set_kthread_struct+0x70/0x70 [ 6508.224067] ret_from_fork+0x1f/0x30 [ 6512.048147] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6512.052323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6512.083070] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6512.087491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6512.090895] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6512.092623] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6512.100106] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6512.106045] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6512.109754] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6512.115231] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6512.118935] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6512.125085] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6512.133055] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6512.145127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6512.152108] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6512.154771] Call Trace: [ 6512.156070] [ 6512.159155] ? show_regs.cold.9+0x22/0x2f [ 6512.163355] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6512.167068] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6512.174757] ? watchdog+0x40/0x40 [ 6512.176332] ? hrtimer_interrupt+0x128/0x2c0 [ 6512.180406] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6512.182511] ? apic_timer_interrupt+0xf/0x20 [ 6512.185975] [ 6512.187219] ? native_safe_halt+0x2e/0x50 [ 6512.189713] kvm_wait+0xcd/0xe0 [ 6512.191781] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6512.195011] ? dump_stack+0xf6/0x10e [ 6512.197183] do_raw_spin_lock+0x177/0x190 [ 6512.200055] _raw_spin_lock+0x21/0x30 [ 6512.202055] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6512.205299] obd_mod_free+0x46/0x140 [ptlrpc] [ 6512.208241] mdc_commit_open+0x91/0xb0 [mdc] [ 6512.210161] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6512.212667] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6512.215039] after_reply+0xe2c/0x1780 [ptlrpc] [ 6512.217832] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6512.220231] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6512.224253] ? do_wait_intr+0xf0/0xf0 [ 6512.226608] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6512.232007] kthread+0x1d1/0x200 [ 6512.234954] ? set_kthread_struct+0x70/0x70 [ 6512.236913] ret_from_fork+0x1f/0x30 [ 6514.143619] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4489s! [ 6514.149023] Showing busy workqueues and worker pools: [ 6514.153070] workqueue events: flags=0x0 [ 6514.156460] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6514.158871] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6514.163073] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6514.166005] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6514.170223] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6514.173371] pending: kfree_rcu_monitor [ 6514.174880] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6514.177729] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6514.182759] workqueue events_long: flags=0x0 [ 6514.183856] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6514.185247] pending: lu_objects_flush [obdclass] [ 6514.186768] workqueue events_unbound: flags=0x2 [ 6514.188068] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6514.190820] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6514.195075] workqueue events_power_efficient: flags=0x80 [ 6514.197085] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6514.199507] pending: neigh_periodic_work, check_lifetime [ 6514.202145] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6514.204949] pending: do_cache_clean, neigh_periodic_work [ 6514.206972] workqueue mm_percpu_wq: flags=0x8 [ 6514.208431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6514.210333] pending: lru_add_drain_per_cpu [ 6514.212028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6514.214958] pending: vmstat_update, lru_add_drain_per_cpu [ 6514.217344] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6514.220139] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6514.222635] workqueue writeback: flags=0x4a [ 6514.223784] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6514.225193] pending: wb_workfn, wb_workfn [ 6514.226252] workqueue xprtiod: flags=0xa [ 6514.227190] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6514.228701] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6514.230192] workqueue ipv6_addrconf: flags=0x40008 [ 6514.232188] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6514.234743] pending: addrconf_verify_work [ 6514.236265] workqueue knbd0-recv: flags=0x1a [ 6514.238010] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6514.240560] in-flight: 655:recv_work [ 6514.242061] workqueue ptlrpc_pinger: flags=0x2 [ 6514.243834] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6514.246018] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6514.248153] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4489s workers=4 idle: 36396 38 11188 [ 6514.251201] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6504s workers=2 idle: 658 [ 6520.053414] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 6520.057675] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6520.089065] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6520.095986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6520.100070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6520.102431] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6520.112429] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6520.119003] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6520.122065] RDX: 00000000000012a5 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6520.125076] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6520.129768] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6520.132986] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6520.137722] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6520.143940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6520.146561] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6520.149766] Call Trace: [ 6520.150605] [ 6520.151392] ? show_regs.cold.9+0x22/0x2f [ 6520.152920] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6520.155159] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6520.157009] ? watchdog+0x40/0x40 [ 6520.158495] ? hrtimer_interrupt+0x128/0x2c0 [ 6520.160369] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6520.162464] ? apic_timer_interrupt+0xf/0x20 [ 6520.164472] [ 6520.165369] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6520.169376] do_raw_spin_lock+0x177/0x190 [ 6520.171458] _raw_spin_lock+0x21/0x30 [ 6520.173391] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6520.179725] ? do_raw_spin_unlock+0x75/0x190 [ 6520.181858] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6520.184885] ? do_wait_intr+0xf0/0xf0 [ 6520.187078] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6520.192494] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6520.194370] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6520.202792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6520.207952] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6520.212066] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6520.217135] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6520.218726] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6520.222008] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6520.225553] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6520.230608] lmv_intent_open+0x338/0x1150 [lmv] [ 6520.233621] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6520.236757] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6520.240498] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6520.243396] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6520.249598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6520.251583] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6520.255433] ? do_raw_spin_unlock+0x75/0x190 [ 6520.258004] ? _raw_spin_unlock+0x12/0x30 [ 6520.261286] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6520.268581] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6520.270402] ll_atomic_open+0x36a/0xd70 [lustre] [ 6520.273094] lookup_open+0x356/0xb40 [ 6520.275234] do_last+0x313/0xfc0 [ 6520.276320] ? nd_jump_root+0xe5/0x160 [ 6520.277777] ? path_init+0x437/0x520 [ 6520.280486] path_openat+0xf7/0x500 [ 6520.283858] ? do_raw_spin_unlock+0x75/0x190 [ 6520.285904] do_filp_open+0x99/0x140 [ 6520.287657] ? getname_flags+0x6e/0x330 [ 6520.289165] ? __check_object_size+0xff/0x256 [ 6520.291148] ? do_raw_spin_unlock+0x75/0x190 [ 6520.293083] ? _raw_spin_unlock+0x12/0x30 [ 6520.294787] do_sys_openat2+0x2b4/0x410 [ 6520.296273] do_sys_open+0x73/0xa0 [ 6520.297625] __x64_sys_openat+0x24/0x30 [ 6520.299033] do_syscall_64+0xc1/0x440 [ 6520.300058] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6520.301901] RIP: 0033:0x7fdc0afbf332 [ 6520.303272] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6520.309993] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6520.312369] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6520.314855] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6520.317087] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6520.319758] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6520.322646] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6524.055416] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 6524.063486] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6524.097111] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6524.101167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6524.106959] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6524.112707] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6524.120084] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6524.126680] RAX: 0000000000002787 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6524.131031] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6524.133123] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6524.137032] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6524.141526] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6524.143785] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6524.146550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6524.147975] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6524.153704] Call Trace: [ 6524.154741] [ 6524.155302] ? show_regs.cold.9+0x22/0x2f [ 6524.157492] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6524.159182] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6524.160397] ? watchdog+0x40/0x40 [ 6524.161397] ? hrtimer_interrupt+0x128/0x2c0 [ 6524.162536] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6524.164931] ? apic_timer_interrupt+0xf/0x20 [ 6524.166605] [ 6524.167309] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6524.172335] do_raw_spin_lock+0x177/0x190 [ 6524.173612] _raw_spin_lock+0x21/0x30 [ 6524.175104] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6524.180143] ? do_raw_spin_unlock+0x75/0x190 [ 6524.182436] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6524.185790] ? do_wait_intr+0xf0/0xf0 [ 6524.186970] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6524.188987] mdc_close+0x53b/0xd70 [mdc] [ 6524.196881] lmv_close+0x18d/0x490 [lmv] [ 6524.198008] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6524.200449] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6524.202108] ll_md_close+0x274/0x800 [lustre] [ 6524.206229] ll_file_release+0x128/0xc50 [lustre] [ 6524.207638] __fput+0x10d/0x440 [ 6524.208583] ____fput+0x12/0x20 [ 6524.209586] task_work_run+0xc6/0x110 [ 6524.211127] exit_to_usermode_loop+0x1dd/0x1f0 [ 6524.212825] do_syscall_64+0x3d6/0x440 [ 6524.213911] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6524.215689] RIP: 0033:0x7fdc0afbfcd8 [ 6524.218024] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6524.229744] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6524.234173] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6524.239600] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6524.243983] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6524.246108] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6524.249588] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6536.051024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6536.053790] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6536.095294] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6536.102430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6536.107705] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6536.117597] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6536.124667] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6536.139404] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6536.143134] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6536.146715] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6536.151886] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6536.155244] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6536.158181] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6536.168552] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6536.174867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6536.178542] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6536.181556] Call Trace: [ 6536.182392] [ 6536.183192] ? show_regs.cold.9+0x22/0x2f [ 6536.184564] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6536.187201] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6536.189700] ? watchdog+0x40/0x40 [ 6536.191936] ? hrtimer_interrupt+0x128/0x2c0 [ 6536.193984] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6536.196995] ? apic_timer_interrupt+0xf/0x20 [ 6536.201463] [ 6536.202203] ? native_safe_halt+0x2e/0x50 [ 6536.206705] kvm_wait+0xcd/0xe0 [ 6536.208854] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6536.214553] do_raw_spin_lock+0x177/0x190 [ 6536.220508] _raw_spin_lock+0x21/0x30 [ 6536.222936] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6536.228565] process_one_work+0x2c8/0x700 [ 6536.231236] worker_thread+0x296/0x6e0 [ 6536.233311] ? rescuer_thread+0x570/0x570 [ 6536.238250] kthread+0x1d1/0x200 [ 6536.239661] ? set_kthread_struct+0x70/0x70 [ 6536.242305] ret_from_fork+0x1f/0x30 [ 6540.048240] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6540.051913] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6540.079289] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6540.084174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6540.088004] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6540.090526] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6540.104903] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6540.109188] RAX: 0000000000000464 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6540.114178] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6540.118552] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6540.122043] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6540.125430] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6540.130418] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6540.135834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6540.138391] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6540.142420] Call Trace: [ 6540.143371] [ 6540.144260] ? show_regs.cold.9+0x22/0x2f [ 6540.146536] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6540.151296] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6540.153104] ? watchdog+0x40/0x40 [ 6540.154124] ? hrtimer_interrupt+0x128/0x2c0 [ 6540.157473] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6540.161132] ? apic_timer_interrupt+0xf/0x20 [ 6540.163295] [ 6540.164365] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6540.166560] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6540.169482] ? dump_stack+0xf6/0x10e [ 6540.172590] do_raw_spin_lock+0x177/0x190 [ 6540.174410] _raw_spin_lock+0x21/0x30 [ 6540.176032] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6540.177904] obd_mod_free+0x46/0x140 [ptlrpc] [ 6540.180459] mdc_commit_open+0x91/0xb0 [mdc] [ 6540.182474] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6540.185791] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6540.190127] after_reply+0xe2c/0x1780 [ptlrpc] [ 6540.192341] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6540.195012] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6540.197451] ? do_wait_intr+0xf0/0xf0 [ 6540.199243] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6540.203467] kthread+0x1d1/0x200 [ 6540.204767] ? set_kthread_struct+0x70/0x70 [ 6540.206692] ret_from_fork+0x1f/0x30 [ 6544.351468] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4519s! [ 6544.367155] Showing busy workqueues and worker pools: [ 6544.370427] workqueue events: flags=0x0 [ 6544.371806] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6544.379919] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6544.391331] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6544.396917] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6544.407763] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6544.413708] pending: kfree_rcu_monitor [ 6544.417011] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6544.420676] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6544.434480] workqueue events_long: flags=0x0 [ 6544.437130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6544.441004] pending: lu_objects_flush [obdclass] [ 6544.444270] workqueue events_unbound: flags=0x2 [ 6544.449033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6544.456058] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6544.466110] workqueue events_power_efficient: flags=0x80 [ 6544.469124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6544.472864] pending: neigh_periodic_work, check_lifetime [ 6544.477585] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6544.482843] pending: do_cache_clean, neigh_periodic_work [ 6544.486064] workqueue mm_percpu_wq: flags=0x8 [ 6544.488374] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6544.491414] pending: lru_add_drain_per_cpu [ 6544.493712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6544.496778] pending: vmstat_update, lru_add_drain_per_cpu [ 6544.499741] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6544.503721] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6544.506011] workqueue writeback: flags=0x4a [ 6544.508844] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6544.516660] pending: wb_workfn, wb_workfn [ 6544.520847] workqueue xprtiod: flags=0xa [ 6544.524114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6544.527738] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6544.531789] workqueue ipv6_addrconf: flags=0x40008 [ 6544.533128] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6544.542779] pending: addrconf_verify_work [ 6544.546846] workqueue knbd0-recv: flags=0x1a [ 6544.550372] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6544.554676] in-flight: 655:recv_work [ 6544.556517] workqueue ptlrpc_pinger: flags=0x2 [ 6544.558847] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6544.562052] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6544.565156] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4519s workers=4 idle: 36396 38 11188 [ 6544.571060] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6535s workers=2 idle: 658 [ 6548.052043] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 6548.054799] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6548.081275] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6548.089620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6548.093373] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6548.095775] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6548.103062] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6548.105936] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6548.108218] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6548.110042] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6548.112661] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6548.115420] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6548.118036] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6548.121346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6548.123511] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6548.126067] Call Trace: [ 6548.126998] [ 6548.127585] ? show_regs.cold.9+0x22/0x2f [ 6548.129566] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6548.130942] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6548.132189] ? watchdog+0x40/0x40 [ 6548.133141] ? hrtimer_interrupt+0x128/0x2c0 [ 6548.134256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6548.135689] ? apic_timer_interrupt+0xf/0x20 [ 6548.137385] [ 6548.138360] ? native_safe_halt+0x2e/0x50 [ 6548.140031] kvm_wait+0xcd/0xe0 [ 6548.141450] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6548.143772] do_raw_spin_lock+0x177/0x190 [ 6548.146168] _raw_spin_lock+0x21/0x30 [ 6548.147750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6548.150372] ? do_raw_spin_unlock+0x75/0x190 [ 6548.152472] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6548.154667] ? do_wait_intr+0xf0/0xf0 [ 6548.156172] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6548.158501] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6548.160942] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6548.163915] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6548.171757] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6548.177554] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6548.180429] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6548.183966] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6548.186553] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6548.190857] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6548.202150] lmv_intent_open+0x338/0x1150 [lmv] [ 6548.206896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6548.209640] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6548.212938] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6548.215593] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6548.217888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6548.220145] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6548.222176] ? do_raw_spin_unlock+0x75/0x190 [ 6548.227613] ? _raw_spin_unlock+0x12/0x30 [ 6548.230078] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6548.239400] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6548.245327] ll_atomic_open+0x36a/0xd70 [lustre] [ 6548.249715] lookup_open+0x356/0xb40 [ 6548.252606] do_last+0x313/0xfc0 [ 6548.255611] ? nd_jump_root+0xe5/0x160 [ 6548.257670] ? path_init+0x437/0x520 [ 6548.263958] path_openat+0xf7/0x500 [ 6548.265180] ? do_raw_spin_unlock+0x75/0x190 [ 6548.268833] do_filp_open+0x99/0x140 [ 6548.272156] ? getname_flags+0x6e/0x330 [ 6548.274135] ? __check_object_size+0xff/0x256 [ 6548.281005] ? do_raw_spin_unlock+0x75/0x190 [ 6548.286468] ? _raw_spin_unlock+0x12/0x30 [ 6548.292491] do_sys_openat2+0x2b4/0x410 [ 6548.297704] do_sys_open+0x73/0xa0 [ 6548.300284] __x64_sys_openat+0x24/0x30 [ 6548.301465] do_syscall_64+0xc1/0x440 [ 6548.302444] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6548.304115] RIP: 0033:0x7fdc0afbf332 [ 6548.314137] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6548.325384] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6548.330713] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6548.335539] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6548.339278] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6548.344260] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6548.351881] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6552.054166] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 6552.062884] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6552.105289] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6552.115306] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6552.123689] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6552.125870] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6552.147313] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6552.154943] RAX: 00000000000048de RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6552.162313] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6552.167832] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6552.174883] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6552.179986] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6552.186251] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6552.193351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6552.199349] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6552.202934] Call Trace: [ 6552.214566] [ 6552.216257] ? show_regs.cold.9+0x22/0x2f [ 6552.222249] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6552.229080] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6552.234496] ? watchdog+0x40/0x40 [ 6552.235664] ? hrtimer_interrupt+0x128/0x2c0 [ 6552.242776] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6552.244424] ? apic_timer_interrupt+0xf/0x20 [ 6552.246302] [ 6552.247030] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6552.250754] do_raw_spin_lock+0x177/0x190 [ 6552.254422] _raw_spin_lock+0x21/0x30 [ 6552.256610] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6552.259262] ? do_raw_spin_unlock+0x75/0x190 [ 6552.262989] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6552.267067] ? do_wait_intr+0xf0/0xf0 [ 6552.268338] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6552.271257] mdc_close+0x53b/0xd70 [mdc] [ 6552.272560] lmv_close+0x18d/0x490 [lmv] [ 6552.281155] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6552.282928] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6552.286847] ll_md_close+0x274/0x800 [lustre] [ 6552.292955] ll_file_release+0x128/0xc50 [lustre] [ 6552.298219] __fput+0x10d/0x440 [ 6552.301713] ____fput+0x12/0x20 [ 6552.305699] task_work_run+0xc6/0x110 [ 6552.310168] exit_to_usermode_loop+0x1dd/0x1f0 [ 6552.312267] do_syscall_64+0x3d6/0x440 [ 6552.316910] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6552.324071] RIP: 0033:0x7fdc0afbfcd8 [ 6552.325222] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6552.337378] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6552.341954] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6552.344129] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6552.347369] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6552.355711] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6552.360009] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6564.051012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6564.054768] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6564.084839] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6564.092305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6564.096579] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6564.104400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6564.112580] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6564.131132] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6564.134296] RAX: 0000000000003af1 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6564.138662] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6564.141802] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6564.145848] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6564.155834] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6564.163734] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6564.172670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6564.176224] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6564.177641] Call Trace: [ 6564.180618] [ 6564.182179] ? show_regs.cold.9+0x22/0x2f [ 6564.184800] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6564.187887] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6564.189976] ? watchdog+0x40/0x40 [ 6564.192136] ? hrtimer_interrupt+0x128/0x2c0 [ 6564.194663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6564.199470] ? apic_timer_interrupt+0xf/0x20 [ 6564.202667] [ 6564.203872] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6564.206923] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6564.210089] do_raw_spin_lock+0x177/0x190 [ 6564.215083] _raw_spin_lock+0x21/0x30 [ 6564.221612] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6564.228548] process_one_work+0x2c8/0x700 [ 6564.229821] worker_thread+0x296/0x6e0 [ 6564.235669] ? rescuer_thread+0x570/0x570 [ 6564.244684] kthread+0x1d1/0x200 [ 6564.252584] ? set_kthread_struct+0x70/0x70 [ 6564.258871] ret_from_fork+0x1f/0x30 [ 6572.048040] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6572.052295] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6572.115923] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6572.130961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6572.143046] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6572.151583] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6572.169865] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6572.176148] RAX: 0000000000005d86 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6572.190730] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6572.197874] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6572.207097] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6572.215707] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6572.225698] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6572.240258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6572.249715] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6572.257370] Call Trace: [ 6572.261198] [ 6572.263125] ? show_regs.cold.9+0x22/0x2f [ 6572.268661] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6572.270991] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6572.274738] ? watchdog+0x40/0x40 [ 6572.276533] ? hrtimer_interrupt+0x128/0x2c0 [ 6572.278777] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6572.281264] ? apic_timer_interrupt+0xf/0x20 [ 6572.284342] [ 6572.285485] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6572.290211] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6572.293576] ? dump_stack+0xf6/0x10e [ 6572.295646] do_raw_spin_lock+0x177/0x190 [ 6572.297340] _raw_spin_lock+0x21/0x30 [ 6572.300823] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6572.304693] obd_mod_free+0x46/0x140 [ptlrpc] [ 6572.307257] mdc_commit_open+0x91/0xb0 [mdc] [ 6572.309736] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6572.312223] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6572.322593] after_reply+0xe2c/0x1780 [ptlrpc] [ 6572.325482] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6572.331263] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6572.334211] ? do_wait_intr+0xf0/0xf0 [ 6572.337814] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6572.341567] kthread+0x1d1/0x200 [ 6572.344506] ? set_kthread_struct+0x70/0x70 [ 6572.348263] ret_from_fork+0x1f/0x30 [ 6575.075100] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4550s! [ 6575.086292] Showing busy workqueues and worker pools: [ 6575.092542] workqueue events: flags=0x0 [ 6575.096804] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6575.104092] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6575.110363] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6575.113259] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6575.118166] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6575.121293] pending: kfree_rcu_monitor [ 6575.123094] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6575.126165] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6575.130947] workqueue events_long: flags=0x0 [ 6575.132511] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6575.136537] pending: lu_objects_flush [obdclass] [ 6575.139443] workqueue events_unbound: flags=0x2 [ 6575.144552] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6575.148771] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6575.154388] workqueue events_power_efficient: flags=0x80 [ 6575.157207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6575.161126] pending: neigh_periodic_work, check_lifetime [ 6575.164181] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6575.167933] pending: do_cache_clean, neigh_periodic_work [ 6575.170763] workqueue mm_percpu_wq: flags=0x8 [ 6575.172906] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6575.176044] pending: lru_add_drain_per_cpu [ 6575.177457] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6575.180257] pending: vmstat_update, lru_add_drain_per_cpu [ 6575.183117] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6575.187166] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6575.189956] workqueue writeback: flags=0x4a [ 6575.192242] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6575.195456] pending: wb_workfn, wb_workfn [ 6575.198813] workqueue xprtiod: flags=0xa [ 6575.200799] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6575.203855] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6575.210461] workqueue ipv6_addrconf: flags=0x40008 [ 6575.212440] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6575.217110] pending: addrconf_verify_work [ 6575.223666] workqueue knbd0-recv: flags=0x1a [ 6575.226913] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6575.230124] in-flight: 655:recv_work [ 6575.232464] workqueue ptlrpc_pinger: flags=0x2 [ 6575.234411] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6575.238105] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6575.240758] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4550s workers=4 idle: 36396 38 11188 [ 6575.244886] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6565s workers=2 idle: 658 [ 6576.053328] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6576.057234] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6576.092228] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6576.101684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6576.106819] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6576.109599] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6576.121200] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6576.125001] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6576.131224] RDX: 0000000000004cef RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6576.136004] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6576.137998] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6576.140691] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6576.144669] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6576.147422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6576.152665] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6576.158490] Call Trace: [ 6576.159796] [ 6576.162302] ? show_regs.cold.9+0x22/0x2f [ 6576.163339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6576.165507] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6576.167766] ? watchdog+0x40/0x40 [ 6576.169273] ? hrtimer_interrupt+0x128/0x2c0 [ 6576.170523] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6576.171619] ? apic_timer_interrupt+0xf/0x20 [ 6576.172874] [ 6576.173342] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6576.181464] do_raw_spin_lock+0x177/0x190 [ 6576.183209] _raw_spin_lock+0x21/0x30 [ 6576.185065] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6576.189119] ? do_raw_spin_unlock+0x75/0x190 [ 6576.196007] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6576.198758] ? do_wait_intr+0xf0/0xf0 [ 6576.200069] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6576.206571] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6576.209385] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6576.211573] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6576.220229] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6576.223196] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6576.226031] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6576.229747] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6576.231587] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6576.235024] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6576.237008] lmv_intent_open+0x338/0x1150 [lmv] [ 6576.238283] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6576.241201] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6576.245238] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6576.250036] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6576.254484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6576.259820] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6576.262328] ? do_raw_spin_unlock+0x75/0x190 [ 6576.263804] ? _raw_spin_unlock+0x12/0x30 [ 6576.265102] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6576.267186] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6576.269305] ll_atomic_open+0x36a/0xd70 [lustre] [ 6576.271262] lookup_open+0x356/0xb40 [ 6576.272798] do_last+0x313/0xfc0 [ 6576.274097] ? nd_jump_root+0xe5/0x160 [ 6576.275697] ? path_init+0x437/0x520 [ 6576.277430] path_openat+0xf7/0x500 [ 6576.278978] ? do_raw_spin_unlock+0x75/0x190 [ 6576.280823] do_filp_open+0x99/0x140 [ 6576.282244] ? getname_flags+0x6e/0x330 [ 6576.283969] ? __check_object_size+0xff/0x256 [ 6576.285856] ? do_raw_spin_unlock+0x75/0x190 [ 6576.289252] ? _raw_spin_unlock+0x12/0x30 [ 6576.294605] do_sys_openat2+0x2b4/0x410 [ 6576.297194] do_sys_open+0x73/0xa0 [ 6576.301074] __x64_sys_openat+0x24/0x30 [ 6576.304125] do_syscall_64+0xc1/0x440 [ 6576.305548] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6576.308612] RIP: 0033:0x7fdc0afbf332 [ 6576.310199] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6576.319123] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6576.324238] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6576.326460] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6576.329101] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6576.332067] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6576.336544] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6579.443017] rcu: INFO: rcu_sched self-detected stall on CPU [ 6579.445979] rcu: 0-...!: (2762919 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 6579.451093] (t=4560100 jiffies g=270761 q=3756) [ 6579.453222] rcu: rcu_sched kthread starved for 4555032 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 6579.458424] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6579.462793] rcu: RCU grace-period kthread stack dump: [ 6579.464846] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6579.469368] Call Trace: [ 6579.470644] __schedule+0x351/0xcb0 [ 6579.471948] schedule+0xc0/0x180 [ 6579.473111] schedule_timeout+0xb4/0x190 [ 6579.474698] ? do_raw_spin_unlock+0x75/0x190 [ 6579.476421] ? __next_timer_interrupt+0x160/0x160 [ 6579.478834] ? prepare_to_swait_event+0xf3/0x140 [ 6579.481065] rcu_gp_kthread+0x8dd/0x15b0 [ 6579.483200] ? force_qs_rnp+0x2c0/0x2c0 [ 6579.484876] kthread+0x1d1/0x200 [ 6579.486320] ? set_kthread_struct+0x70/0x70 [ 6579.488239] ret_from_fork+0x1f/0x30 [ 6579.490322] rcu: Stack dump where RCU GP kthread last ran: [ 6579.493047] Sending NMI from CPU 0 to CPUs 3: [ 6579.496817] NMI backtrace for cpu 3 [ 6579.496827] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6579.496832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6579.496834] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6579.496901] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6579.496904] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 [ 6579.496906] RAX: 00000000000031f2 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6579.496908] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6579.496909] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6579.496911] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6579.496915] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6579.496917] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6579.496918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6579.496920] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6579.496923] Call Trace: [ 6579.496924] [ 6579.496925] ? show_regs.cold.9+0x22/0x2f [ 6579.496927] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6579.496928] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6579.496929] ? nmi_handle+0xbd/0x210 [ 6579.496930] ? default_do_nmi+0x4b/0x1b0 [ 6579.496932] ? do_nmi+0x280/0x380 [ 6579.496933] ? end_repeat_nmi+0x16/0x1f [ 6579.496934] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6579.496935] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6579.496936] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6579.496940] [ 6579.496941] do_raw_spin_lock+0x177/0x190 [ 6579.496943] _raw_spin_lock+0x21/0x30 [ 6579.496944] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6579.496945] ? do_raw_spin_unlock+0x75/0x190 [ 6579.496946] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6579.496947] ? do_wait_intr+0xf0/0xf0 [ 6579.496948] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6579.496952] mdc_close+0x53b/0xd70 [mdc] [ 6579.496953] lmv_close+0x18d/0x490 [lmv] [ 6579.496954] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6579.496956] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6579.496957] ll_md_close+0x274/0x800 [lustre] [ 6579.496958] ll_file_release+0x128/0xc50 [lustre] [ 6579.496960] __fput+0x10d/0x440 [ 6579.496961] ____fput+0x12/0x20 [ 6579.496962] task_work_run+0xc6/0x110 [ 6579.496963] exit_to_usermode_loop+0x1dd/0x1f0 [ 6579.496964] do_syscall_64+0x3d6/0x440 [ 6579.496965] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6579.496966] RIP: 0033:0x7fdc0afbfcd8 [ 6579.496968] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6579.496970] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6579.496975] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6579.496977] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6579.496978] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6579.496980] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6579.496981] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6579.498194] NMI backtrace for cpu 0 [ 6579.654727] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6579.660066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6579.663038] Call Trace: [ 6579.663989] [ 6579.665684] ? dump_stack+0xbb/0x10e [ 6579.666945] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6579.669878] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6579.672231] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6579.673959] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6579.675759] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6579.677857] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6579.679207] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6579.682048] ? update_process_times+0x6d/0xc0 [ 6579.683800] ? tick_sched_handle.isra.18+0x27/0x90 [ 6579.688530] ? tick_sched_timer+0x53/0xb0 [ 6579.691041] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6579.692951] ? tick_sched_handle.isra.18+0x90/0x90 [ 6579.696788] ? hrtimer_interrupt+0x128/0x2c0 [ 6579.699078] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6579.705825] ? apic_timer_interrupt+0xf/0x20 [ 6579.708031] [ 6579.708896] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6579.712136] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6579.717751] ? dump_stack+0xf6/0x10e [ 6579.718970] ? do_raw_spin_lock+0x177/0x190 [ 6579.724613] ? _raw_spin_lock+0x21/0x30 [ 6579.727038] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6579.731203] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6579.734734] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6579.742891] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6579.744839] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6579.746737] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6579.749495] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6579.755209] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6579.759990] ? do_wait_intr+0xf0/0xf0 [ 6579.761630] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6579.763527] ? kthread+0x1d1/0x200 [ 6579.765539] ? set_kthread_struct+0x70/0x70 [ 6579.766933] ? ret_from_fork+0x1f/0x30 [ 6592.055597] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6592.060802] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6592.112018] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6592.129577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6592.143971] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6592.150543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6592.155415] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6592.164484] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6592.169534] RAX: 0000000000005887 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6592.176747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6592.188747] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6592.193495] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6592.202912] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6592.206011] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6592.211701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6592.215892] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6592.226486] Call Trace: [ 6592.227292] [ 6592.227943] ? show_regs.cold.9+0x22/0x2f [ 6592.229245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6592.246323] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6592.254408] ? watchdog+0x40/0x40 [ 6592.255514] ? hrtimer_interrupt+0x128/0x2c0 [ 6592.256864] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6592.268210] ? apic_timer_interrupt+0xf/0x20 [ 6592.269611] [ 6592.270309] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6592.276855] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6592.285590] do_raw_spin_lock+0x177/0x190 [ 6592.289458] _raw_spin_lock+0x21/0x30 [ 6592.291283] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6592.295596] process_one_work+0x2c8/0x700 [ 6592.299457] worker_thread+0x296/0x6e0 [ 6592.301153] ? rescuer_thread+0x570/0x570 [ 6592.302702] kthread+0x1d1/0x200 [ 6592.305559] ? set_kthread_struct+0x70/0x70 [ 6592.307749] ret_from_fork+0x1f/0x30 [ 6604.048043] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6604.051324] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6604.052019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6604.061410] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6604.061425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6604.061518] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6604.061524] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6604.061528] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6604.061543] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6604.061548] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6604.061553] RAX: 0000000000007b97 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6604.061556] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6604.061559] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6604.061561] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6604.061564] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6604.061568] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6604.061572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6604.061575] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6604.061584] Call Trace: [ 6604.061589] [ 6604.061595] ? show_regs.cold.9+0x22/0x2f [ 6604.061604] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6604.061610] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6604.061615] ? watchdog+0x40/0x40 [ 6604.061619] ? hrtimer_interrupt+0x128/0x2c0 [ 6604.061624] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6604.061634] ? apic_timer_interrupt+0xf/0x20 [ 6604.061638] [ 6604.061641] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6604.061649] do_raw_spin_lock+0x177/0x190 [ 6604.061654] _raw_spin_lock+0x21/0x30 [ 6604.061659] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6604.061935] ? do_raw_spin_unlock+0x75/0x190 [ 6604.061941] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6604.062143] ? do_wait_intr+0xf0/0xf0 [ 6604.062151] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6604.062360] mdc_close+0x53b/0xd70 [mdc] [ 6604.062397] lmv_close+0x18d/0x490 [lmv] [ 6604.062434] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6604.062554] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6604.062634] ll_md_close+0x274/0x800 [lustre] [ 6604.062717] ll_file_release+0x128/0xc50 [lustre] [ 6604.062799] __fput+0x10d/0x440 [ 6604.062808] ____fput+0x12/0x20 [ 6604.062813] task_work_run+0xc6/0x110 [ 6604.062821] exit_to_usermode_loop+0x1dd/0x1f0 [ 6604.062828] do_syscall_64+0x3d6/0x440 [ 6604.062833] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6604.062840] RIP: 0033:0x7fdc0afbfcd8 [ 6604.062847] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6604.062851] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6604.062856] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6604.062859] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6604.062861] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6604.062864] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6604.062866] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6604.140217] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6604.146592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6604.146657] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6604.146670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6604.146673] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6604.146686] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6604.146690] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6604.146693] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6604.146695] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6604.146696] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6604.146698] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6604.146699] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6604.146702] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6604.146704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6604.146706] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6604.146714] Call Trace: [ 6604.146720] [ 6604.146726] ? show_regs.cold.9+0x22/0x2f [ 6604.146736] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6604.146742] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6604.146746] ? watchdog+0x40/0x40 [ 6604.146751] ? hrtimer_interrupt+0x128/0x2c0 [ 6604.146755] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6604.146763] ? apic_timer_interrupt+0xf/0x20 [ 6604.146768] [ 6604.146771] ? native_safe_halt+0x2e/0x50 [ 6604.146775] kvm_wait+0xcd/0xe0 [ 6604.146782] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6604.146796] do_raw_spin_lock+0x177/0x190 [ 6604.146801] _raw_spin_lock+0x21/0x30 [ 6604.146806] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6604.147058] ? do_raw_spin_unlock+0x75/0x190 [ 6604.147064] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6604.147207] ? do_wait_intr+0xf0/0xf0 [ 6604.147215] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6604.152265] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6604.152277] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6604.152292] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6604.152296] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6604.152303] RAX: 0000000000006786 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6604.169747] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6604.170067] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6604.170211] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6604.170321] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6604.170354] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6604.170373] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6604.170390] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6604.170445] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6604.170586] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6604.170604] lmv_intent_open+0x338/0x1150 [lmv] [ 6604.170625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6604.170704] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6604.170759] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6604.170783] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6604.170910] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6604.170964] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6604.171018] ? do_raw_spin_unlock+0x75/0x190 [ 6604.171024] ? _raw_spin_unlock+0x12/0x30 [ 6604.171030] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6604.171050] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6604.171062] ll_atomic_open+0x36a/0xd70 [lustre] [ 6604.171118] lookup_open+0x356/0xb40 [ 6604.171126] do_last+0x313/0xfc0 [ 6604.171129] ? nd_jump_root+0xe5/0x160 [ 6604.171133] ? path_init+0x437/0x520 [ 6604.171136] path_openat+0xf7/0x500 [ 6604.171140] ? do_raw_spin_unlock+0x75/0x190 [ 6604.171143] do_filp_open+0x99/0x140 [ 6604.171147] ? getname_flags+0x6e/0x330 [ 6604.171151] ? __check_object_size+0xff/0x256 [ 6604.171156] ? do_raw_spin_unlock+0x75/0x190 [ 6604.171159] ? _raw_spin_unlock+0x12/0x30 [ 6604.171161] do_sys_openat2+0x2b4/0x410 [ 6604.171164] do_sys_open+0x73/0xa0 [ 6604.171167] __x64_sys_openat+0x24/0x30 [ 6604.171169] do_syscall_64+0xc1/0x440 [ 6604.171174] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6604.171179] RIP: 0033:0x7fdc0afbf332 [ 6604.171185] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6604.171188] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6604.171192] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6604.171193] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6604.171195] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6604.171196] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6604.171198] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6604.730940] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6604.738558] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6604.745739] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6604.751204] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6604.758356] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6604.766788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6604.773925] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6604.780037] Call Trace: [ 6604.783925] [ 6604.785791] ? show_regs.cold.9+0x22/0x2f [ 6604.789622] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6604.794305] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6604.795920] ? watchdog+0x40/0x40 [ 6604.800265] ? hrtimer_interrupt+0x128/0x2c0 [ 6604.802036] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6604.809431] ? apic_timer_interrupt+0xf/0x20 [ 6604.818028] [ 6604.818789] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6604.832023] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6604.834053] ? dump_stack+0xf6/0x10e [ 6604.838886] do_raw_spin_lock+0x177/0x190 [ 6604.842878] _raw_spin_lock+0x21/0x30 [ 6604.849152] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6604.851033] obd_mod_free+0x46/0x140 [ptlrpc] [ 6604.864914] mdc_commit_open+0x91/0xb0 [mdc] [ 6604.869077] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6604.876653] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6604.887192] after_reply+0xe2c/0x1780 [ptlrpc] [ 6604.895009] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6604.900375] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6604.912934] ? do_wait_intr+0xf0/0xf0 [ 6604.916985] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6604.922983] kthread+0x1d1/0x200 [ 6604.925373] ? set_kthread_struct+0x70/0x70 [ 6604.929382] ret_from_fork+0x1f/0x30 [ 6605.281517] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4580s! [ 6605.308958] Showing busy workqueues and worker pools: [ 6605.316516] workqueue events: flags=0x0 [ 6605.325814] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6605.343305] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6605.349666] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6605.358442] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6605.372379] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6605.375415] pending: kfree_rcu_monitor [ 6605.382353] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6605.401383] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6605.415977] workqueue events_long: flags=0x0 [ 6605.432939] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6605.442600] pending: lu_objects_flush [obdclass] [ 6605.447397] workqueue events_unbound: flags=0x2 [ 6605.450842] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6605.456382] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6605.468183] workqueue events_power_efficient: flags=0x80 [ 6605.480639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6605.490689] pending: neigh_periodic_work, check_lifetime [ 6605.495631] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6605.501764] pending: do_cache_clean, neigh_periodic_work [ 6605.511332] workqueue mm_percpu_wq: flags=0x8 [ 6605.522491] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6605.535636] pending: lru_add_drain_per_cpu [ 6605.542646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6605.551775] pending: vmstat_update, lru_add_drain_per_cpu [ 6605.560218] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6605.564290] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6605.577262] workqueue writeback: flags=0x4a [ 6605.592687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6605.594988] pending: wb_workfn, wb_workfn [ 6605.604464] workqueue xprtiod: flags=0xa [ 6605.613862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6605.625358] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6605.636195] workqueue ipv6_addrconf: flags=0x40008 [ 6605.641108] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6605.649946] pending: addrconf_verify_work [ 6605.655075] workqueue knbd0-recv: flags=0x1a [ 6605.658041] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6605.662434] in-flight: 655:recv_work [ 6605.664473] workqueue ptlrpc_pinger: flags=0x2 [ 6605.667158] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6605.670376] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6605.678778] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4580s workers=4 idle: 36396 38 11188 [ 6605.686334] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6596s workers=2 idle: 658 [ 6620.052183] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6620.058671] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6620.126948] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6620.146997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6620.152882] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6620.165766] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6620.167787] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6620.202876] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6620.207558] RAX: 000000000000648e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6620.223972] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6620.241504] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6620.252955] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6620.260031] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6620.267676] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6620.286502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6620.290552] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6620.302733] Call Trace: [ 6620.304137] [ 6620.311532] ? show_regs.cold.9+0x22/0x2f [ 6620.324117] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6620.337690] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6620.347890] ? watchdog+0x40/0x40 [ 6620.356929] ? hrtimer_interrupt+0x128/0x2c0 [ 6620.358383] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6620.372047] ? apic_timer_interrupt+0xf/0x20 [ 6620.391259] [ 6620.394019] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6620.404072] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6620.408398] do_raw_spin_lock+0x177/0x190 [ 6620.410851] _raw_spin_lock+0x21/0x30 [ 6620.413894] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6620.425935] process_one_work+0x2c8/0x700 [ 6620.433028] worker_thread+0x296/0x6e0 [ 6620.442674] ? rescuer_thread+0x570/0x570 [ 6620.456009] kthread+0x1d1/0x200 [ 6620.457715] ? set_kthread_struct+0x70/0x70 [ 6620.465755] ret_from_fork+0x1f/0x30 [ 6632.048012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6632.054830] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6632.055442] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6632.055606] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6632.055613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6632.055616] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6632.055630] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6632.055635] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6632.055639] RAX: 0000000000007014 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6632.055642] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6632.055645] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6632.055647] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6632.055649] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6632.055652] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6632.055655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6632.055657] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6632.055666] Call Trace: [ 6632.055672] [ 6632.055678] ? show_regs.cold.9+0x22/0x2f [ 6632.055687] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6632.059872] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6632.059878] ? watchdog+0x40/0x40 [ 6632.059882] ? hrtimer_interrupt+0x128/0x2c0 [ 6632.059886] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6632.059899] ? apic_timer_interrupt+0xf/0x20 [ 6632.059904] [ 6632.059909] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6632.059915] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6632.059918] ? dump_stack+0xf6/0x10e [ 6632.059925] do_raw_spin_lock+0x177/0x190 [ 6632.059930] _raw_spin_lock+0x21/0x30 [ 6632.059939] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6632.060784] obd_mod_free+0x46/0x140 [ptlrpc] [ 6632.061199] mdc_commit_open+0x91/0xb0 [mdc] [ 6632.061468] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6632.061481] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6632.061560] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6632.061566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6632.061568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6632.061580] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6632.061584] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6632.061588] RAX: 0000000000002852 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6632.061591] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6632.061593] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6632.061595] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6632.061597] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6632.061601] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6632.061604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6632.061607] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6632.061615] Call Trace: [ 6632.061619] [ 6632.061622] ? show_regs.cold.9+0x22/0x2f [ 6632.061630] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6632.061635] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6632.061639] ? watchdog+0x40/0x40 [ 6632.061643] ? hrtimer_interrupt+0x128/0x2c0 [ 6632.061647] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6632.061653] ? apic_timer_interrupt+0xf/0x20 [ 6632.061658] [ 6632.061661] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6632.061667] do_raw_spin_lock+0x177/0x190 [ 6632.061671] _raw_spin_lock+0x21/0x30 [ 6632.061245] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6632.061676] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6632.063578] ? do_raw_spin_unlock+0x75/0x190 [ 6632.063586] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6632.063994] ? do_wait_intr+0xf0/0xf0 [ 6632.064004] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6632.064947] mdc_close+0x53b/0xd70 [mdc] [ 6632.064989] lmv_close+0x18d/0x490 [lmv] [ 6632.065020] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6632.067013] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6632.067784] ll_md_close+0x274/0x800 [lustre] [ 6632.067868] ll_file_release+0x128/0xc50 [lustre] [ 6632.067950] __fput+0x10d/0x440 [ 6632.067960] ____fput+0x12/0x20 [ 6632.067965] task_work_run+0xc6/0x110 [ 6632.067974] exit_to_usermode_loop+0x1dd/0x1f0 [ 6632.067981] do_syscall_64+0x3d6/0x440 [ 6632.067985] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6632.067994] RIP: 0033:0x7fdc0afbfcd8 [ 6632.068047] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6632.068052] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6632.068057] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6632.068060] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6632.068062] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6632.068064] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6632.068067] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6632.071470] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6632.071559] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6632.071568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6632.071570] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6632.071585] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6632.071589] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6632.071594] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6632.071596] RDX: 0000000000007f4b RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6632.071598] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6632.071601] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6632.071603] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6632.071606] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6632.071610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6632.071612] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6632.071622] Call Trace: [ 6632.071628] [ 6632.071634] ? show_regs.cold.9+0x22/0x2f [ 6632.071644] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6632.071650] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6632.071654] ? watchdog+0x40/0x40 [ 6632.071659] ? hrtimer_interrupt+0x128/0x2c0 [ 6632.071663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6632.071672] ? apic_timer_interrupt+0xf/0x20 [ 6632.071677] [ 6632.071680] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6632.071686] do_raw_spin_lock+0x177/0x190 [ 6632.071703] _raw_spin_lock+0x21/0x30 [ 6632.073163] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6632.074875] ? do_raw_spin_unlock+0x75/0x190 [ 6632.074888] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6632.076621] ? do_wait_intr+0xf0/0xf0 [ 6632.076635] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6632.148245] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6632.148966] after_reply+0xe2c/0x1780 [ptlrpc] [ 6632.149968] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6632.176619] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6632.177981] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6632.191407] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6632.198875] ? do_wait_intr+0xf0/0xf0 [ 6632.198901] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6632.199756] kthread+0x1d1/0x200 [ 6632.199763] ? set_kthread_struct+0x70/0x70 [ 6632.199767] ret_from_fork+0x1f/0x30 [ 6633.403446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6633.408040] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6633.411659] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6633.420327] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6633.425553] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6633.432608] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6633.438271] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6633.448407] lmv_intent_open+0x338/0x1150 [lmv] [ 6633.453386] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6633.457441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6633.460406] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6633.462132] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6633.469560] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6633.480662] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6633.489311] ? do_raw_spin_unlock+0x75/0x190 [ 6633.495074] ? _raw_spin_unlock+0x12/0x30 [ 6633.506401] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6633.514583] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6633.522945] ll_atomic_open+0x36a/0xd70 [lustre] [ 6633.527633] lookup_open+0x356/0xb40 [ 6633.530196] do_last+0x313/0xfc0 [ 6633.535615] ? nd_jump_root+0xe5/0x160 [ 6633.536841] ? path_init+0x437/0x520 [ 6633.550580] path_openat+0xf7/0x500 [ 6633.556061] ? do_raw_spin_unlock+0x75/0x190 [ 6633.559654] do_filp_open+0x99/0x140 [ 6633.561364] ? getname_flags+0x6e/0x330 [ 6633.564983] ? __check_object_size+0xff/0x256 [ 6633.567774] ? do_raw_spin_unlock+0x75/0x190 [ 6633.570829] ? _raw_spin_unlock+0x12/0x30 [ 6633.574957] do_sys_openat2+0x2b4/0x410 [ 6633.576571] do_sys_open+0x73/0xa0 [ 6633.578949] __x64_sys_openat+0x24/0x30 [ 6633.580878] do_syscall_64+0xc1/0x440 [ 6633.583005] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6633.586375] RIP: 0033:0x7fdc0afbf332 [ 6633.589814] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6633.609965] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6633.616816] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6633.621826] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6633.626892] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6633.631423] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6633.634933] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6635.999119] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4611s! [ 6636.007451] Showing busy workqueues and worker pools: [ 6636.011627] workqueue events: flags=0x0 [ 6636.015670] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6636.023375] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6636.040657] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6636.052885] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6636.062155] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6636.071752] pending: kfree_rcu_monitor [ 6636.074049] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6636.078235] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6636.086588] workqueue events_long: flags=0x0 [ 6636.089973] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6636.093705] pending: lu_objects_flush [obdclass] [ 6636.109586] workqueue events_unbound: flags=0x2 [ 6636.111748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6636.114319] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6636.120846] workqueue events_power_efficient: flags=0x80 [ 6636.123766] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6636.126564] pending: neigh_periodic_work, check_lifetime [ 6636.130605] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6636.133911] pending: do_cache_clean, neigh_periodic_work [ 6636.136577] workqueue mm_percpu_wq: flags=0x8 [ 6636.138639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6636.141543] pending: lru_add_drain_per_cpu [ 6636.143459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6636.146342] pending: vmstat_update, lru_add_drain_per_cpu [ 6636.148566] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6636.150494] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6636.153473] workqueue writeback: flags=0x4a [ 6636.155037] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6636.159401] pending: wb_workfn, wb_workfn [ 6636.162084] workqueue xprtiod: flags=0xa [ 6636.163915] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6636.167244] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6636.171256] workqueue ipv6_addrconf: flags=0x40008 [ 6636.173484] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6636.176744] pending: addrconf_verify_work [ 6636.178838] workqueue knbd0-recv: flags=0x1a [ 6636.180924] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6636.183956] in-flight: 655:recv_work [ 6636.185788] workqueue ptlrpc_pinger: flags=0x2 [ 6636.187899] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6636.190858] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6636.193144] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4611s workers=4 idle: 36396 38 11188 [ 6636.196159] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6626s workers=2 idle: 658 [ 6648.051417] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6648.056840] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6648.079562] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6648.085467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6648.089469] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6648.093193] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6648.096039] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6648.106774] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6648.111375] RAX: 0000000000007471 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6648.114349] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6648.118325] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6648.121963] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6648.125597] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6648.129352] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6648.134615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6648.143118] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6648.147994] Call Trace: [ 6648.149016] [ 6648.150155] ? show_regs.cold.9+0x22/0x2f [ 6648.152447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6648.154803] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6648.157321] ? watchdog+0x40/0x40 [ 6648.158839] ? hrtimer_interrupt+0x128/0x2c0 [ 6648.161040] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6648.163533] ? apic_timer_interrupt+0xf/0x20 [ 6648.165555] [ 6648.166698] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6648.169606] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6648.171942] do_raw_spin_lock+0x177/0x190 [ 6648.173821] _raw_spin_lock+0x21/0x30 [ 6648.175937] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6648.178709] process_one_work+0x2c8/0x700 [ 6648.180594] worker_thread+0x296/0x6e0 [ 6648.182074] ? rescuer_thread+0x570/0x570 [ 6648.183722] kthread+0x1d1/0x200 [ 6648.185603] ? set_kthread_struct+0x70/0x70 [ 6648.187924] ret_from_fork+0x1f/0x30 [ 6660.055027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6660.057591] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6660.065297] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6660.065386] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6660.065395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6660.065398] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6660.065414] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6660.065418] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6660.065423] RAX: 0000000000004bac RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6660.065425] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6660.065428] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6660.065430] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6660.065432] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6660.065435] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6660.065438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6660.065440] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6660.065450] Call Trace: [ 6660.065455] [ 6660.065461] ? show_regs.cold.9+0x22/0x2f [ 6660.065470] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6660.065478] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6660.065482] ? watchdog+0x40/0x40 [ 6660.065487] ? hrtimer_interrupt+0x128/0x2c0 [ 6660.065490] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6660.065499] ? apic_timer_interrupt+0xf/0x20 [ 6660.065503] [ 6660.065506] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6660.065513] do_raw_spin_lock+0x177/0x190 [ 6660.065518] _raw_spin_lock+0x21/0x30 [ 6660.065523] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6660.065849] ? do_raw_spin_unlock+0x75/0x190 [ 6660.065855] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6660.066052] ? do_wait_intr+0xf0/0xf0 [ 6660.066060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6660.066269] mdc_close+0x53b/0xd70 [mdc] [ 6660.066309] lmv_close+0x18d/0x490 [lmv] [ 6660.066742] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6660.066880] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6660.066969] ll_md_close+0x274/0x800 [lustre] [ 6660.067056] ll_file_release+0x128/0xc50 [lustre] [ 6660.067147] __fput+0x10d/0x440 [ 6660.067223] ____fput+0x12/0x20 [ 6660.067229] task_work_run+0xc6/0x110 [ 6660.067244] exit_to_usermode_loop+0x1dd/0x1f0 [ 6660.067262] do_syscall_64+0x3d6/0x440 [ 6660.067267] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6660.067274] RIP: 0033:0x7fdc0afbfcd8 [ 6660.067282] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6660.067286] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6660.067290] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6660.067293] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6660.067295] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6660.067298] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6660.067300] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6660.256923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6660.273750] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6660.277261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6660.279973] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6660.281828] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6660.287478] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6660.289815] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6660.293485] RDX: 00000000000009e7 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6660.301702] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6660.306966] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6660.313007] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6660.317985] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6660.324045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6660.329012] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6660.333073] Call Trace: [ 6660.334680] [ 6660.335668] ? show_regs.cold.9+0x22/0x2f [ 6660.340312] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6660.343061] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6660.345812] ? watchdog+0x40/0x40 [ 6660.347825] ? hrtimer_interrupt+0x128/0x2c0 [ 6660.350127] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6660.357094] ? apic_timer_interrupt+0xf/0x20 [ 6660.360261] [ 6660.361271] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6660.368136] do_raw_spin_lock+0x177/0x190 [ 6660.370937] _raw_spin_lock+0x21/0x30 [ 6660.375950] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6660.380543] ? do_raw_spin_unlock+0x75/0x190 [ 6660.383353] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6660.386988] ? do_wait_intr+0xf0/0xf0 [ 6660.389405] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6660.392024] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6660.396215] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6660.400066] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6660.406920] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6660.413372] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6660.416103] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6660.420815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6660.429425] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6660.437318] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6660.445443] lmv_intent_open+0x338/0x1150 [lmv] [ 6660.448036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6660.453226] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6660.463305] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6660.471879] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6660.484309] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6660.492726] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6660.499926] ? do_raw_spin_unlock+0x75/0x190 [ 6660.502977] ? _raw_spin_unlock+0x12/0x30 [ 6660.507874] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6660.515007] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6660.520142] ll_atomic_open+0x36a/0xd70 [lustre] [ 6660.527302] lookup_open+0x356/0xb40 [ 6660.529962] do_last+0x313/0xfc0 [ 6660.534596] ? nd_jump_root+0xe5/0x160 [ 6660.536001] ? path_init+0x437/0x520 [ 6660.538916] path_openat+0xf7/0x500 [ 6660.541003] ? do_raw_spin_unlock+0x75/0x190 [ 6660.544398] do_filp_open+0x99/0x140 [ 6660.547045] ? getname_flags+0x6e/0x330 [ 6660.555360] ? __check_object_size+0xff/0x256 [ 6660.563008] ? do_raw_spin_unlock+0x75/0x190 [ 6660.573737] ? _raw_spin_unlock+0x12/0x30 [ 6660.579050] do_sys_openat2+0x2b4/0x410 [ 6660.580326] do_sys_open+0x73/0xa0 [ 6660.585932] __x64_sys_openat+0x24/0x30 [ 6660.588991] do_syscall_64+0xc1/0x440 [ 6660.590824] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6660.594967] RIP: 0033:0x7fdc0afbf332 [ 6660.596158] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6660.622700] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6660.638570] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6660.654642] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6660.661164] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6660.670148] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6660.685607] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6664.053329] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6664.064854] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6664.138520] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6664.150696] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6664.161141] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6664.164532] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6664.179757] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6664.184201] RAX: 0000000000000238 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6664.187960] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6664.191691] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6664.195586] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6664.199729] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6664.203417] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6664.207475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6664.211241] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6664.214976] Call Trace: [ 6664.216338] [ 6664.217644] ? show_regs.cold.9+0x22/0x2f [ 6664.219517] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6664.221126] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6664.222796] ? watchdog+0x40/0x40 [ 6664.224459] ? hrtimer_interrupt+0x128/0x2c0 [ 6664.226468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6664.228747] ? apic_timer_interrupt+0xf/0x20 [ 6664.231578] [ 6664.233280] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6664.236392] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6664.239949] ? dump_stack+0xf6/0x10e [ 6664.242413] do_raw_spin_lock+0x177/0x190 [ 6664.245256] _raw_spin_lock+0x21/0x30 [ 6664.247391] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6664.250453] obd_mod_free+0x46/0x140 [ptlrpc] [ 6664.252974] mdc_commit_open+0x91/0xb0 [mdc] [ 6664.254580] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6664.257781] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6664.261403] after_reply+0xe2c/0x1780 [ptlrpc] [ 6664.263991] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6664.266473] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6664.269296] ? do_wait_intr+0xf0/0xf0 [ 6664.272030] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6664.275629] kthread+0x1d1/0x200 [ 6664.277462] ? set_kthread_struct+0x70/0x70 [ 6664.280265] ret_from_fork+0x1f/0x30 [ 6666.207085] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4641s! [ 6666.210701] Showing busy workqueues and worker pools: [ 6666.214245] workqueue events: flags=0x0 [ 6666.218159] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6666.223364] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6666.227473] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6666.233090] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6666.240472] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6666.248177] pending: kfree_rcu_monitor [ 6666.250195] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6666.253373] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6666.258716] workqueue events_long: flags=0x0 [ 6666.260663] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6666.263600] pending: lu_objects_flush [obdclass] [ 6666.266379] workqueue events_unbound: flags=0x2 [ 6666.268537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6666.272525] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6666.277345] workqueue events_power_efficient: flags=0x80 [ 6666.279957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6666.282643] pending: neigh_periodic_work, check_lifetime [ 6666.285410] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6666.288389] pending: do_cache_clean, neigh_periodic_work [ 6666.291238] workqueue mm_percpu_wq: flags=0x8 [ 6666.292858] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6666.296218] pending: lru_add_drain_per_cpu [ 6666.298699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6666.302497] pending: vmstat_update, lru_add_drain_per_cpu [ 6666.305625] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6666.310240] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6666.318335] workqueue writeback: flags=0x4a [ 6666.321432] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6666.328809] pending: wb_workfn, wb_workfn [ 6666.330017] workqueue xprtiod: flags=0xa [ 6666.331165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6666.335756] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6666.342590] workqueue ipv6_addrconf: flags=0x40008 [ 6666.346392] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6666.349480] pending: addrconf_verify_work [ 6666.351506] workqueue knbd0-recv: flags=0x1a [ 6666.353614] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6666.357651] in-flight: 655:recv_work [ 6666.360019] workqueue ptlrpc_pinger: flags=0x2 [ 6666.362124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6666.365267] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6666.368025] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4641s workers=4 idle: 36396 38 11188 [ 6666.374074] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6656s workers=2 idle: 658 [ 6676.051044] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6676.056291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6676.094793] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6676.101646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6676.108646] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6676.112507] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6676.115242] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6676.135363] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6676.142909] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6676.151091] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6676.161242] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6676.170539] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6676.178986] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6676.186557] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6676.197957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6676.204595] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6676.213604] Call Trace: [ 6676.214535] [ 6676.215129] ? show_regs.cold.9+0x22/0x2f [ 6676.238293] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6676.250917] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6676.256945] ? watchdog+0x40/0x40 [ 6676.258169] ? hrtimer_interrupt+0x128/0x2c0 [ 6676.267227] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6676.269987] ? apic_timer_interrupt+0xf/0x20 [ 6676.274797] [ 6676.276261] ? native_safe_halt+0x2e/0x50 [ 6676.278443] kvm_wait+0xcd/0xe0 [ 6676.280344] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6676.286904] do_raw_spin_lock+0x177/0x190 [ 6676.291915] _raw_spin_lock+0x21/0x30 [ 6676.295387] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6676.303230] process_one_work+0x2c8/0x700 [ 6676.307289] worker_thread+0x296/0x6e0 [ 6676.311061] ? rescuer_thread+0x570/0x570 [ 6676.313850] kthread+0x1d1/0x200 [ 6676.315659] ? set_kthread_struct+0x70/0x70 [ 6676.318547] ret_from_fork+0x1f/0x30 [ 6688.052278] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6688.056454] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6688.058037] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6688.096399] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6688.100851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6688.101015] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6688.101023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6688.101027] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6688.101041] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6688.101047] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6688.101051] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6688.101053] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6688.101056] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6688.101058] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6688.101061] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6688.101064] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6688.101067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6688.101069] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6688.101078] Call Trace: [ 6688.101084] [ 6688.101090] ? show_regs.cold.9+0x22/0x2f [ 6688.101099] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6688.101106] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6688.101111] ? watchdog+0x40/0x40 [ 6688.101115] ? hrtimer_interrupt+0x128/0x2c0 [ 6688.101119] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6688.101126] ? apic_timer_interrupt+0xf/0x20 [ 6688.101131] [ 6688.101135] ? native_safe_halt+0x2e/0x50 [ 6688.101139] kvm_wait+0xcd/0xe0 [ 6688.101147] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6688.101155] do_raw_spin_lock+0x177/0x190 [ 6688.101160] _raw_spin_lock+0x21/0x30 [ 6688.101164] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6688.109245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6688.109256] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6688.109272] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6688.109277] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6688.109282] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6688.109285] RDX: 0000000000000405 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6688.109287] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6688.109290] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6688.109292] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6688.109296] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6688.109299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6688.109301] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6688.109311] Call Trace: [ 6688.109317] [ 6688.109323] ? show_regs.cold.9+0x22/0x2f [ 6688.109334] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6688.109340] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6688.109345] ? watchdog+0x40/0x40 [ 6688.109350] ? hrtimer_interrupt+0x128/0x2c0 [ 6688.109354] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6688.109362] ? apic_timer_interrupt+0xf/0x20 [ 6688.109367] [ 6688.109371] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6688.109376] do_raw_spin_lock+0x177/0x190 [ 6688.109380] _raw_spin_lock+0x21/0x30 [ 6688.109388] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6688.109852] ? do_raw_spin_unlock+0x75/0x190 [ 6688.109858] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6688.110062] ? do_wait_intr+0xf0/0xf0 [ 6688.110070] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6688.110265] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6688.142245] ? do_raw_spin_unlock+0x75/0x190 [ 6688.142268] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6688.149859] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6688.150169] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6688.150294] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6688.150338] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6688.150369] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6688.150396] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6688.150479] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6688.150884] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6688.150928] lmv_intent_open+0x338/0x1150 [lmv] [ 6688.150959] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6688.151063] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6688.151147] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6688.151173] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6688.151261] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6688.158270] ? do_wait_intr+0xf0/0xf0 [ 6688.158291] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6688.158571] mdc_close+0x53b/0xd70 [mdc] [ 6688.158608] lmv_close+0x18d/0x490 [lmv] [ 6688.158639] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6688.158750] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6688.158847] ll_md_close+0x274/0x800 [lustre] [ 6688.159070] ll_file_release+0x128/0xc50 [lustre] [ 6688.159154] __fput+0x10d/0x440 [ 6688.159164] ____fput+0x12/0x20 [ 6688.159168] task_work_run+0xc6/0x110 [ 6688.159176] exit_to_usermode_loop+0x1dd/0x1f0 [ 6688.159183] do_syscall_64+0x3d6/0x440 [ 6688.159188] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6688.159199] RIP: 0033:0x7fdc0afbfcd8 [ 6688.159207] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6688.159211] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6688.159217] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6688.159220] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6688.159222] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6688.159224] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6688.159227] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6688.452093] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6688.454612] ? do_raw_spin_unlock+0x75/0x190 [ 6688.456283] ? _raw_spin_unlock+0x12/0x30 [ 6688.461629] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6688.464552] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6688.467809] ll_atomic_open+0x36a/0xd70 [lustre] [ 6688.469655] lookup_open+0x356/0xb40 [ 6688.473305] do_last+0x313/0xfc0 [ 6688.474645] ? nd_jump_root+0xe5/0x160 [ 6688.481194] ? path_init+0x437/0x520 [ 6688.482536] path_openat+0xf7/0x500 [ 6688.484265] ? do_raw_spin_unlock+0x75/0x190 [ 6688.489838] do_filp_open+0x99/0x140 [ 6688.491021] ? getname_flags+0x6e/0x330 [ 6688.492012] ? __check_object_size+0xff/0x256 [ 6688.496581] ? do_raw_spin_unlock+0x75/0x190 [ 6688.498230] ? _raw_spin_unlock+0x12/0x30 [ 6688.499929] do_sys_openat2+0x2b4/0x410 [ 6688.504523] do_sys_open+0x73/0xa0 [ 6688.506853] __x64_sys_openat+0x24/0x30 [ 6688.511465] do_syscall_64+0xc1/0x440 [ 6688.514515] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6688.518015] RIP: 0033:0x7fdc0afbf332 [ 6688.519181] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6688.534892] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6688.538602] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6688.540480] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6688.543577] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6688.547211] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6688.553817] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6692.048940] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6692.051978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6692.073264] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6692.078048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6692.081451] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6692.084819] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6692.095254] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6692.099656] RAX: 000000000000743d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6692.103166] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6692.106062] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6692.109304] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6692.112497] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6692.116023] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6692.119430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6692.122459] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6692.125404] Call Trace: [ 6692.126519] [ 6692.127219] ? show_regs.cold.9+0x22/0x2f [ 6692.129377] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6692.131472] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6692.133279] ? watchdog+0x40/0x40 [ 6692.134868] ? hrtimer_interrupt+0x128/0x2c0 [ 6692.137664] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6692.139422] ? apic_timer_interrupt+0xf/0x20 [ 6692.141154] [ 6692.142647] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6692.145038] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6692.147547] ? dump_stack+0xf6/0x10e [ 6692.149007] do_raw_spin_lock+0x177/0x190 [ 6692.151089] _raw_spin_lock+0x21/0x30 [ 6692.153036] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6692.155624] obd_mod_free+0x46/0x140 [ptlrpc] [ 6692.158428] mdc_commit_open+0x91/0xb0 [mdc] [ 6692.160442] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6692.163850] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6692.167351] after_reply+0xe2c/0x1780 [ptlrpc] [ 6692.170021] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6692.173467] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6692.175759] ? do_wait_intr+0xf0/0xf0 [ 6692.177619] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6692.179873] kthread+0x1d1/0x200 [ 6692.181212] ? set_kthread_struct+0x70/0x70 [ 6692.183249] ret_from_fork+0x1f/0x30 [ 6696.416407] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4671s! [ 6696.419711] Showing busy workqueues and worker pools: [ 6696.421699] workqueue events: flags=0x0 [ 6696.424069] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6696.426223] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6696.434756] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6696.438117] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6696.447025] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6696.450281] pending: kfree_rcu_monitor [ 6696.452292] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6696.459542] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6696.471549] workqueue events_long: flags=0x0 [ 6696.474165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6696.482913] pending: lu_objects_flush [obdclass] [ 6696.485577] workqueue events_unbound: flags=0x2 [ 6696.487690] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6696.490429] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6696.494484] workqueue events_power_efficient: flags=0x80 [ 6696.498578] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6696.507897] pending: neigh_periodic_work, check_lifetime [ 6696.514419] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6696.518092] pending: do_cache_clean, neigh_periodic_work [ 6696.523311] workqueue mm_percpu_wq: flags=0x8 [ 6696.528446] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6696.530548] pending: lru_add_drain_per_cpu [ 6696.533198] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6696.535912] pending: vmstat_update, lru_add_drain_per_cpu [ 6696.538395] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6696.541149] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6696.543120] workqueue writeback: flags=0x4a [ 6696.544558] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6696.547085] pending: wb_workfn, wb_workfn [ 6696.551103] workqueue xprtiod: flags=0xa [ 6696.555518] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6696.557572] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6696.566372] workqueue ipv6_addrconf: flags=0x40008 [ 6696.568273] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6696.571134] pending: addrconf_verify_work [ 6696.572857] workqueue knbd0-recv: flags=0x1a [ 6696.575051] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6696.580326] in-flight: 655:recv_work [ 6696.582908] workqueue ptlrpc_pinger: flags=0x2 [ 6696.584696] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6696.592561] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6696.595355] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4671s workers=4 idle: 36396 38 11188 [ 6696.598186] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6687s workers=2 idle: 658 [ 6704.051353] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6704.055034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6704.081436] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6704.085587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6704.088417] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6704.090849] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6704.092682] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6704.103407] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6704.106498] RAX: 00000000000061f4 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6704.108881] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6704.111183] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6704.113393] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6704.116159] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6704.118502] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6704.121406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6704.122735] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6704.124621] Call Trace: [ 6704.125468] [ 6704.126139] ? show_regs.cold.9+0x22/0x2f [ 6704.127435] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6704.129438] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6704.131718] ? watchdog+0x40/0x40 [ 6704.133260] ? hrtimer_interrupt+0x128/0x2c0 [ 6704.135013] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6704.137178] ? apic_timer_interrupt+0xf/0x20 [ 6704.138843] [ 6704.139672] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6704.141685] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6704.143942] do_raw_spin_lock+0x177/0x190 [ 6704.145371] _raw_spin_lock+0x21/0x30 [ 6704.148428] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6704.151624] process_one_work+0x2c8/0x700 [ 6704.154636] worker_thread+0x296/0x6e0 [ 6704.156517] ? rescuer_thread+0x570/0x570 [ 6704.166762] kthread+0x1d1/0x200 [ 6704.168695] ? set_kthread_struct+0x70/0x70 [ 6704.169899] ret_from_fork+0x1f/0x30 [ 6716.053478] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6716.054028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6716.057861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6716.057946] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6716.057954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6716.057957] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6716.057971] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6716.057975] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6716.057980] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6716.057983] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6716.057986] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6716.057988] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6716.057991] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6716.057994] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6716.057998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6716.058000] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6716.058009] Call Trace: [ 6716.058014] [ 6716.058021] ? show_regs.cold.9+0x22/0x2f [ 6716.058031] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6716.058037] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6716.058041] ? watchdog+0x40/0x40 [ 6716.058045] ? hrtimer_interrupt+0x128/0x2c0 [ 6716.058048] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6716.058054] ? apic_timer_interrupt+0xf/0x20 [ 6716.058059] [ 6716.058061] ? native_safe_halt+0x2e/0x50 [ 6716.058066] kvm_wait+0xcd/0xe0 [ 6716.058073] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6716.058080] do_raw_spin_lock+0x177/0x190 [ 6716.058085] _raw_spin_lock+0x21/0x30 [ 6716.058090] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6716.058389] ? do_raw_spin_unlock+0x75/0x190 [ 6716.058395] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6716.058615] ? do_wait_intr+0xf0/0xf0 [ 6716.058625] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6716.058840] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6716.059073] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6716.065951] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6716.066043] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6716.066053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6716.066055] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6716.066070] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6716.066074] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6716.066079] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6716.066081] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6716.066083] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6716.066086] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6716.066088] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6716.066091] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6716.066094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6716.066096] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6716.066106] Call Trace: [ 6716.066111] [ 6716.066118] ? show_regs.cold.9+0x22/0x2f [ 6716.066254] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6716.066455] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6716.066460] ? watchdog+0x40/0x40 [ 6716.066464] ? hrtimer_interrupt+0x128/0x2c0 [ 6716.066469] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6716.066475] ? apic_timer_interrupt+0xf/0x20 [ 6716.066480] [ 6716.066483] ? native_safe_halt+0x2e/0x50 [ 6716.066487] kvm_wait+0xcd/0xe0 [ 6716.066495] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6716.066506] do_raw_spin_lock+0x177/0x190 [ 6716.066511] _raw_spin_lock+0x21/0x30 [ 6716.066516] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6716.066833] ? do_raw_spin_unlock+0x75/0x190 [ 6716.066839] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6716.067362] ? do_wait_intr+0xf0/0xf0 [ 6716.067370] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6716.067574] mdc_close+0x53b/0xd70 [mdc] [ 6716.067611] lmv_close+0x18d/0x490 [lmv] [ 6716.067638] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6716.067745] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6716.067826] ll_md_close+0x274/0x800 [lustre] [ 6716.067981] ll_file_release+0x128/0xc50 [lustre] [ 6716.068066] __fput+0x10d/0x440 [ 6716.068076] ____fput+0x12/0x20 [ 6716.068080] task_work_run+0xc6/0x110 [ 6716.068088] exit_to_usermode_loop+0x1dd/0x1f0 [ 6716.068094] do_syscall_64+0x3d6/0x440 [ 6716.068099] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6716.068106] RIP: 0033:0x7fdc0afbfcd8 [ 6716.068112] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6716.068116] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6716.068121] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6716.068210] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6716.068213] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6716.068215] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6716.068217] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6716.526816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6716.531622] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6716.535394] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6716.537973] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6716.540187] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6716.543798] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6716.553053] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6716.562937] lmv_intent_open+0x338/0x1150 [lmv] [ 6716.565769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6716.567411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6716.572779] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6716.576803] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6716.581603] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6716.586845] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6716.592106] ? do_raw_spin_unlock+0x75/0x190 [ 6716.595829] ? _raw_spin_unlock+0x12/0x30 [ 6716.599604] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6716.602353] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6716.604570] ll_atomic_open+0x36a/0xd70 [lustre] [ 6716.607228] lookup_open+0x356/0xb40 [ 6716.609026] do_last+0x313/0xfc0 [ 6716.610823] ? nd_jump_root+0xe5/0x160 [ 6716.611901] ? path_init+0x437/0x520 [ 6716.614519] path_openat+0xf7/0x500 [ 6716.615757] ? do_raw_spin_unlock+0x75/0x190 [ 6716.616922] do_filp_open+0x99/0x140 [ 6716.619594] ? getname_flags+0x6e/0x330 [ 6716.621478] ? __check_object_size+0xff/0x256 [ 6716.624562] ? do_raw_spin_unlock+0x75/0x190 [ 6716.628450] ? _raw_spin_unlock+0x12/0x30 [ 6716.630557] do_sys_openat2+0x2b4/0x410 [ 6716.632596] do_sys_open+0x73/0xa0 [ 6716.634739] __x64_sys_openat+0x24/0x30 [ 6716.638484] do_syscall_64+0xc1/0x440 [ 6716.641514] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6716.646822] RIP: 0033:0x7fdc0afbf332 [ 6716.651440] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6716.664333] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6716.669226] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6716.673023] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6716.679248] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6716.683717] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6716.688079] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6724.048017] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 6724.052783] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6724.080209] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6724.091494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6724.096799] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6724.102664] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6724.120015] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6724.126951] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6724.129388] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6724.134663] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6724.141191] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6724.145146] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6724.150114] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6724.159225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6724.165539] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6724.177664] Call Trace: [ 6724.179385] [ 6724.181193] ? show_regs.cold.9+0x22/0x2f [ 6724.183553] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6724.186236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6724.188831] ? watchdog+0x40/0x40 [ 6724.190682] ? hrtimer_interrupt+0x128/0x2c0 [ 6724.193812] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6724.197554] ? apic_timer_interrupt+0xf/0x20 [ 6724.200058] [ 6724.201259] ? native_safe_halt+0x2e/0x50 [ 6724.203534] kvm_wait+0xcd/0xe0 [ 6724.205338] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6724.208323] ? dump_stack+0xf6/0x10e [ 6724.210565] do_raw_spin_lock+0x177/0x190 [ 6724.212651] _raw_spin_lock+0x21/0x30 [ 6724.215031] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6724.218153] obd_mod_free+0x46/0x140 [ptlrpc] [ 6724.221243] mdc_commit_open+0x91/0xb0 [mdc] [ 6724.223561] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6724.227347] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6724.231564] after_reply+0xe2c/0x1780 [ptlrpc] [ 6724.234197] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6724.236671] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6724.239662] ? do_wait_intr+0xf0/0xf0 [ 6724.242589] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6724.246242] kthread+0x1d1/0x200 [ 6724.248559] ? set_kthread_struct+0x70/0x70 [ 6724.250909] ret_from_fork+0x1f/0x30 [ 6726.626312] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4701s! [ 6726.641265] Showing busy workqueues and worker pools: [ 6726.645216] workqueue events: flags=0x0 [ 6726.648479] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6726.660198] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6726.664827] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6726.669094] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6726.683329] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6726.689740] pending: kfree_rcu_monitor [ 6726.693106] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6726.700786] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6726.709942] workqueue events_long: flags=0x0 [ 6726.717831] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6726.722006] pending: lu_objects_flush [obdclass] [ 6726.724866] workqueue events_unbound: flags=0x2 [ 6726.728068] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6726.732427] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6726.739603] workqueue events_power_efficient: flags=0x80 [ 6726.741378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6726.744408] pending: neigh_periodic_work, check_lifetime [ 6726.747022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6726.756680] pending: do_cache_clean, neigh_periodic_work [ 6726.759394] workqueue mm_percpu_wq: flags=0x8 [ 6726.761424] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6726.764754] pending: lru_add_drain_per_cpu [ 6726.766341] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6726.770292] pending: vmstat_update, lru_add_drain_per_cpu [ 6726.776855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6726.780474] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6726.783800] workqueue writeback: flags=0x4a [ 6726.789347] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6726.791934] pending: wb_workfn, wb_workfn [ 6726.796500] workqueue xprtiod: flags=0xa [ 6726.802101] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6726.804663] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6726.813404] workqueue ipv6_addrconf: flags=0x40008 [ 6726.815730] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6726.819170] pending: addrconf_verify_work [ 6726.824932] workqueue knbd0-recv: flags=0x1a [ 6726.828593] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6726.831701] in-flight: 655:recv_work [ 6726.833598] workqueue ptlrpc_pinger: flags=0x2 [ 6726.835124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6726.842132] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6726.851352] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4701s workers=4 idle: 36396 38 11188 [ 6726.857171] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6717s workers=2 idle: 658 [ 6732.051385] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6732.057812] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6732.075541] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6732.080209] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6732.083552] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6732.085922] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6732.088009] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6732.095033] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6732.097983] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6732.099975] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6732.102204] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6732.104584] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6732.107393] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6732.109174] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6732.111671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6732.113568] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6732.116183] Call Trace: [ 6732.116969] [ 6732.117808] ? show_regs.cold.9+0x22/0x2f [ 6732.119012] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6732.120879] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6732.122327] ? watchdog+0x40/0x40 [ 6732.123271] ? hrtimer_interrupt+0x128/0x2c0 [ 6732.124830] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6732.126163] ? apic_timer_interrupt+0xf/0x20 [ 6732.127616] [ 6732.128189] ? native_safe_halt+0x2e/0x50 [ 6732.129523] kvm_wait+0xcd/0xe0 [ 6732.130565] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6732.132268] do_raw_spin_lock+0x177/0x190 [ 6732.133797] _raw_spin_lock+0x21/0x30 [ 6732.135191] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6732.137439] process_one_work+0x2c8/0x700 [ 6732.138987] worker_thread+0x296/0x6e0 [ 6732.140276] ? rescuer_thread+0x570/0x570 [ 6732.141818] kthread+0x1d1/0x200 [ 6732.142862] ? set_kthread_struct+0x70/0x70 [ 6732.144057] ret_from_fork+0x1f/0x30 [ 6744.052074] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6744.055385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6744.056502] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6744.089454] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6744.089473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6744.089476] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6744.089497] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6744.089505] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6744.089511] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6744.089514] RDX: 0000000000006bcb RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6744.089516] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6744.089518] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6744.089521] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6744.089528] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6744.089531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6744.089537] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6744.089545] Call Trace: [ 6744.089550] [ 6744.089556] ? show_regs.cold.9+0x22/0x2f [ 6744.089566] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6744.089573] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6744.089578] ? watchdog+0x40/0x40 [ 6744.089588] ? hrtimer_interrupt+0x128/0x2c0 [ 6744.089592] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6744.089667] ? apic_timer_interrupt+0xf/0x20 [ 6744.089671] [ 6744.089675] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6744.089680] do_raw_spin_lock+0x177/0x190 [ 6744.089685] _raw_spin_lock+0x21/0x30 [ 6744.089692] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6744.089984] ? do_raw_spin_unlock+0x75/0x190 [ 6744.089990] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6744.095694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 [ 6744.101020] ? do_wait_intr+0xf0/0xf0 [ 6744.101043] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6744.107912] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6744.107951] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6744.107958] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6744.107961] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6744.107975] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6744.107979] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6744.107984] RAX: 00000000000049e4 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6744.108148] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6744.108152] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6744.108154] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6744.108157] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6744.108160] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6744.108163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6744.108165] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6744.108174] Call Trace: [ 6744.108180] [ 6744.108186] ? show_regs.cold.9+0x22/0x2f [ 6744.108196] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6744.108203] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6744.108207] ? watchdog+0x40/0x40 [ 6744.108212] ? hrtimer_interrupt+0x128/0x2c0 [ 6744.108217] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6744.108225] ? apic_timer_interrupt+0xf/0x20 [ 6744.108230] [ 6744.108233] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6744.108240] do_raw_spin_lock+0x177/0x190 [ 6744.108245] _raw_spin_lock+0x21/0x30 [ 6744.108250] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6744.108669] ? do_raw_spin_unlock+0x75/0x190 [ 6744.108676] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6744.108896] ? do_wait_intr+0xf0/0xf0 [ 6744.108905] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6744.114222] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6744.126707] mdc_close+0x53b/0xd70 [mdc] [ 6744.129821] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6744.135462] lmv_close+0x18d/0x490 [lmv] [ 6744.135504] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6744.139699] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6744.139887] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6744.139925] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6744.140074] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6744.140106] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6744.140199] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6744.140602] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6744.140635] lmv_intent_open+0x338/0x1150 [lmv] [ 6744.140670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6744.141856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6744.141951] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6744.141987] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6744.142091] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6744.142189] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6744.142346] ? do_raw_spin_unlock+0x75/0x190 [ 6744.142355] ? _raw_spin_unlock+0x12/0x30 [ 6744.142362] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6744.142461] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6744.142481] ll_atomic_open+0x36a/0xd70 [lustre] [ 6744.142576] lookup_open+0x356/0xb40 [ 6744.142585] do_last+0x313/0xfc0 [ 6744.142590] ? nd_jump_root+0xe5/0x160 [ 6744.142595] ? path_init+0x437/0x520 [ 6744.142600] path_openat+0xf7/0x500 [ 6744.142605] ? do_raw_spin_unlock+0x75/0x190 [ 6744.142610] do_filp_open+0x99/0x140 [ 6744.142618] ? getname_flags+0x6e/0x330 [ 6744.142622] ? __check_object_size+0xff/0x256 [ 6744.142631] ? do_raw_spin_unlock+0x75/0x190 [ 6744.142634] ? _raw_spin_unlock+0x12/0x30 [ 6744.142638] do_sys_openat2+0x2b4/0x410 [ 6744.142643] do_sys_open+0x73/0xa0 [ 6744.142647] __x64_sys_openat+0x24/0x30 [ 6744.142651] do_syscall_64+0xc1/0x440 [ 6744.142657] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6744.142664] RIP: 0033:0x7fdc0afbf332 [ 6744.142671] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6744.142675] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6744.142680] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6744.142682] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6744.142684] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6744.142686] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6744.142688] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6744.587914] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6744.593062] ll_md_close+0x274/0x800 [lustre] [ 6744.595302] ll_file_release+0x128/0xc50 [lustre] [ 6744.603233] __fput+0x10d/0x440 [ 6744.605117] ____fput+0x12/0x20 [ 6744.606483] task_work_run+0xc6/0x110 [ 6744.608870] exit_to_usermode_loop+0x1dd/0x1f0 [ 6744.610703] do_syscall_64+0x3d6/0x440 [ 6744.615594] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6744.621089] RIP: 0033:0x7fdc0afbfcd8 [ 6744.623116] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6744.636973] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6744.642146] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6744.645450] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6744.650160] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6744.653148] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6744.656285] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6752.050694] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 6752.057054] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6752.129652] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6752.137653] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6752.142860] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6752.146003] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6752.159049] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6752.164984] RAX: 0000000000006387 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6752.176153] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6752.182277] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6752.189085] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6752.200090] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6752.209591] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6752.213872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6752.222542] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6752.227867] Call Trace: [ 6752.229241] [ 6752.233817] ? show_regs.cold.9+0x22/0x2f [ 6752.237430] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6752.246045] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6752.249920] ? watchdog+0x40/0x40 [ 6752.252122] ? hrtimer_interrupt+0x128/0x2c0 [ 6752.254822] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6752.257951] ? apic_timer_interrupt+0xf/0x20 [ 6752.259436] [ 6752.263263] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6752.266147] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6752.271266] ? dump_stack+0xf6/0x10e [ 6752.276058] do_raw_spin_lock+0x177/0x190 [ 6752.280566] _raw_spin_lock+0x21/0x30 [ 6752.284209] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6752.286199] obd_mod_free+0x46/0x140 [ptlrpc] [ 6752.290787] mdc_commit_open+0x91/0xb0 [mdc] [ 6752.294513] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6752.298925] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6752.303610] after_reply+0xe2c/0x1780 [ptlrpc] [ 6752.307481] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6752.309629] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6752.312802] ? do_wait_intr+0xf0/0xf0 [ 6752.317119] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6752.321668] kthread+0x1d1/0x200 [ 6752.323604] ? set_kthread_struct+0x70/0x70 [ 6752.324777] ret_from_fork+0x1f/0x30 [ 6757.344932] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4732s! [ 6757.352314] Showing busy workqueues and worker pools: [ 6757.364792] workqueue events: flags=0x0 [ 6757.366202] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6757.383727] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6757.387796] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6757.390725] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6757.394758] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6757.397360] pending: kfree_rcu_monitor [ 6757.399014] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6757.401759] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6757.406543] workqueue events_long: flags=0x0 [ 6757.407932] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6757.410262] pending: lu_objects_flush [obdclass] [ 6757.412376] workqueue events_unbound: flags=0x2 [ 6757.414085] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6757.416575] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6757.420381] workqueue events_power_efficient: flags=0x80 [ 6757.422490] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6757.425094] pending: neigh_periodic_work, check_lifetime [ 6757.427868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6757.430502] pending: do_cache_clean, neigh_periodic_work [ 6757.432792] workqueue mm_percpu_wq: flags=0x8 [ 6757.434245] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6757.436840] pending: lru_add_drain_per_cpu [ 6757.438597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6757.440983] pending: vmstat_update, lru_add_drain_per_cpu [ 6757.443026] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6757.445621] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6757.447503] workqueue writeback: flags=0x4a [ 6757.448795] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6757.451078] pending: wb_workfn, wb_workfn [ 6757.452801] workqueue xprtiod: flags=0xa [ 6757.454429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6757.456249] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6757.461265] workqueue ipv6_addrconf: flags=0x40008 [ 6757.463210] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6757.465765] pending: addrconf_verify_work [ 6757.467466] workqueue knbd0-recv: flags=0x1a [ 6757.468698] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6757.471092] in-flight: 655:recv_work [ 6757.472634] workqueue ptlrpc_pinger: flags=0x2 [ 6757.473974] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6757.475565] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6757.477616] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4732s workers=4 idle: 36396 38 11188 [ 6757.480520] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6748s workers=2 idle: 658 [ 6759.446034] rcu: INFO: rcu_sched self-detected stall on CPU [ 6759.448147] rcu: 0-...!: (2876781 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 6759.452684] (t=4740103 jiffies g=270761 q=3838) [ 6759.454862] rcu: rcu_sched kthread starved for 4735035 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 6759.459691] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6759.466565] rcu: RCU grace-period kthread stack dump: [ 6759.468629] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6759.472279] Call Trace: [ 6759.473186] __schedule+0x351/0xcb0 [ 6759.474629] schedule+0xc0/0x180 [ 6759.475740] schedule_timeout+0xb4/0x190 [ 6759.477250] ? do_raw_spin_unlock+0x75/0x190 [ 6759.478860] ? __next_timer_interrupt+0x160/0x160 [ 6759.480446] ? prepare_to_swait_event+0xf3/0x140 [ 6759.482253] rcu_gp_kthread+0x8dd/0x15b0 [ 6759.484117] ? force_qs_rnp+0x2c0/0x2c0 [ 6759.485386] kthread+0x1d1/0x200 [ 6759.486169] ? set_kthread_struct+0x70/0x70 [ 6759.487904] ret_from_fork+0x1f/0x30 [ 6759.489205] rcu: Stack dump where RCU GP kthread last ran: [ 6759.491344] Sending NMI from CPU 0 to CPUs 2: [ 6759.494994] NMI backtrace for cpu 2 [ 6759.495002] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6759.495004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6759.495006] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6759.495008] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6759.495010] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 [ 6759.495013] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6759.495015] RDX: 00000000000073d4 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6759.495016] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6759.495017] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6759.495019] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6759.495020] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6759.495022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6759.495023] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6759.495024] Call Trace: [ 6759.495026] [ 6759.495026] ? show_regs.cold.9+0x22/0x2f [ 6759.495028] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6759.495029] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6759.495030] ? nmi_handle+0xbd/0x210 [ 6759.495032] ? default_do_nmi+0x4b/0x1b0 [ 6759.495033] ? do_nmi+0x280/0x380 [ 6759.495034] ? end_repeat_nmi+0x16/0x1f [ 6759.495035] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6759.495036] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6759.495037] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6759.495039] [ 6759.495040] do_raw_spin_lock+0x177/0x190 [ 6759.495044] _raw_spin_lock+0x21/0x30 [ 6759.495045] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6759.495047] ? do_raw_spin_unlock+0x75/0x190 [ 6759.495048] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6759.495049] ? do_wait_intr+0xf0/0xf0 [ 6759.495051] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6759.495052] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6759.495053] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6759.495055] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6759.495056] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6759.495057] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6759.495059] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6759.495060] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6759.495064] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6759.495065] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6759.495067] lmv_intent_open+0x338/0x1150 [lmv] [ 6759.495068] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6759.495070] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6759.495071] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6759.495072] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6759.495074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6759.495075] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6759.495076] ? do_raw_spin_unlock+0x75/0x190 [ 6759.495077] ? _raw_spin_unlock+0x12/0x30 [ 6759.495079] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6759.495080] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6759.495081] ll_atomic_open+0x36a/0xd70 [lustre] [ 6759.495082] lookup_open+0x356/0xb40 [ 6759.495083] do_last+0x313/0xfc0 [ 6759.495084] ? nd_jump_root+0xe5/0x160 [ 6759.495085] ? path_init+0x437/0x520 [ 6759.495086] path_openat+0xf7/0x500 [ 6759.495087] ? do_raw_spin_unlock+0x75/0x190 [ 6759.495089] do_filp_open+0x99/0x140 [ 6759.495089] ? getname_flags+0x6e/0x330 [ 6759.495091] ? __check_object_size+0xff/0x256 [ 6759.495092] ? do_raw_spin_unlock+0x75/0x190 [ 6759.495093] ? _raw_spin_unlock+0x12/0x30 [ 6759.495094] do_sys_openat2+0x2b4/0x410 [ 6759.495095] do_sys_open+0x73/0xa0 [ 6759.495096] __x64_sys_openat+0x24/0x30 [ 6759.495097] do_syscall_64+0xc1/0x440 [ 6759.495098] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6759.495099] RIP: 0033:0x7fdc0afbf332 [ 6759.495101] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6759.495104] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6759.495106] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6759.495108] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6759.495109] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6759.495111] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6759.495113] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6759.495185] NMI backtrace for cpu 0 [ 6759.631011] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6759.634941] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6759.637991] Call Trace: [ 6759.639134] [ 6759.640047] ? dump_stack+0xbb/0x10e [ 6759.641311] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6759.643080] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6759.644621] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6759.646398] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6759.648540] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6759.650374] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6759.652788] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6759.654743] ? update_process_times+0x6d/0xc0 [ 6759.656264] ? tick_sched_handle.isra.18+0x27/0x90 [ 6759.657892] ? tick_sched_timer+0x53/0xb0 [ 6759.659609] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6759.661304] ? tick_sched_handle.isra.18+0x90/0x90 [ 6759.663097] ? hrtimer_interrupt+0x128/0x2c0 [ 6759.664444] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6759.665977] ? apic_timer_interrupt+0xf/0x20 [ 6759.667604] [ 6759.668490] ? native_safe_halt+0x2e/0x50 [ 6759.670002] ? kvm_wait+0xcd/0xe0 [ 6759.671296] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6759.673185] ? dump_stack+0xf6/0x10e [ 6759.674483] ? do_raw_spin_lock+0x177/0x190 [ 6759.675859] ? _raw_spin_lock+0x21/0x30 [ 6759.677428] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6759.680935] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6759.685524] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6759.687237] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6759.689299] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6759.691451] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6759.693324] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6759.695355] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6759.697045] ? do_wait_intr+0xf0/0xf0 [ 6759.698204] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6759.700488] ? kthread+0x1d1/0x200 [ 6759.701516] ? set_kthread_struct+0x70/0x70 [ 6759.703280] ? ret_from_fork+0x1f/0x30 [ 6760.053432] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6760.064403] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6760.113487] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6760.125661] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6760.135359] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6760.138362] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6760.141085] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6760.154025] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6760.156577] RAX: 0000000000005197 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6760.159056] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6760.161585] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6760.164577] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6760.167439] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6760.170260] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6760.173398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6760.178337] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6760.180448] Call Trace: [ 6760.181176] [ 6760.184707] ? show_regs.cold.9+0x22/0x2f [ 6760.188874] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6760.191616] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6760.195511] ? watchdog+0x40/0x40 [ 6760.199271] ? hrtimer_interrupt+0x128/0x2c0 [ 6760.200694] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6760.202415] ? apic_timer_interrupt+0xf/0x20 [ 6760.206416] [ 6760.207200] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6760.213407] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6760.215483] do_raw_spin_lock+0x177/0x190 [ 6760.217942] _raw_spin_lock+0x21/0x30 [ 6760.219291] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6760.221674] process_one_work+0x2c8/0x700 [ 6760.226644] worker_thread+0x296/0x6e0 [ 6760.231569] ? rescuer_thread+0x570/0x570 [ 6760.233924] kthread+0x1d1/0x200 [ 6760.235613] ? set_kthread_struct+0x70/0x70 [ 6760.242774] ret_from_fork+0x1f/0x30 [ 6772.054029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6772.062946] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6772.104388] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6772.113963] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6772.124650] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6772.132305] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6772.153915] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6772.158625] RAX: 0000000000005a85 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6772.162698] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6772.167797] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6772.181604] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6772.195220] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6772.202693] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6772.209151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6772.214170] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6772.222986] Call Trace: [ 6772.225082] [ 6772.232428] ? show_regs.cold.9+0x22/0x2f [ 6772.236854] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6772.243259] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6772.248412] ? watchdog+0x40/0x40 [ 6772.252126] ? hrtimer_interrupt+0x128/0x2c0 [ 6772.256690] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6772.260833] ? apic_timer_interrupt+0xf/0x20 [ 6772.265012] [ 6772.267076] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6772.274562] do_raw_spin_lock+0x177/0x190 [ 6772.281754] _raw_spin_lock+0x21/0x30 [ 6772.286189] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6772.292961] ? do_raw_spin_unlock+0x75/0x190 [ 6772.298673] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6772.305164] ? do_wait_intr+0xf0/0xf0 [ 6772.309370] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6772.315393] mdc_close+0x53b/0xd70 [mdc] [ 6772.323515] lmv_close+0x18d/0x490 [lmv] [ 6772.329195] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6772.335021] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6772.340546] ll_md_close+0x274/0x800 [lustre] [ 6772.344495] ll_file_release+0x128/0xc50 [lustre] [ 6772.348949] __fput+0x10d/0x440 [ 6772.352618] ____fput+0x12/0x20 [ 6772.356213] task_work_run+0xc6/0x110 [ 6772.358012] exit_to_usermode_loop+0x1dd/0x1f0 [ 6772.360035] do_syscall_64+0x3d6/0x440 [ 6772.361658] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6772.369124] RIP: 0033:0x7fdc0afbfcd8 [ 6772.374699] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6772.387434] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6772.396816] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6772.403297] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6772.410741] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6772.417597] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6772.423144] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6784.050232] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 6784.052019] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 6784.059451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6784.059533] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6784.059540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6784.059544] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6784.059557] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6784.059561] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6784.059566] RAX: 0000000000007f8b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6784.059569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6784.059571] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6784.059573] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6784.059576] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6784.059579] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6784.059582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6784.059585] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6784.059594] Call Trace: [ 6784.059600] [ 6784.059606] ? show_regs.cold.9+0x22/0x2f [ 6784.063887] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6784.063968] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6784.063975] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6784.063977] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6784.063990] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6784.063994] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6784.063998] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6784.064001] RDX: 0000000000006f31 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6784.064003] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6784.064005] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6784.064007] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6784.064011] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6784.064013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6784.064015] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6784.064025] Call Trace: [ 6784.064030] [ 6784.064037] ? show_regs.cold.9+0x22/0x2f [ 6784.064046] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6784.064052] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6784.064057] ? watchdog+0x40/0x40 [ 6784.064061] ? hrtimer_interrupt+0x128/0x2c0 [ 6784.064065] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6784.064074] ? apic_timer_interrupt+0xf/0x20 [ 6784.064094] [ 6784.064098] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6784.064103] do_raw_spin_lock+0x177/0x190 [ 6784.064108] _raw_spin_lock+0x21/0x30 [ 6784.064115] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6784.064397] ? do_raw_spin_unlock+0x75/0x190 [ 6784.064567] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6784.107525] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6784.120035] ? do_wait_intr+0xf0/0xf0 [ 6784.120058] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6784.120656] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6784.121769] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6784.121988] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6784.142491] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6784.145556] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6784.145662] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6784.145704] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6784.145732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6784.145839] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6784.146327] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6784.146358] lmv_intent_open+0x338/0x1150 [lmv] [ 6784.146390] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6784.146479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6784.146561] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6784.146581] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6784.146663] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6784.146929] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6784.147012] ? do_raw_spin_unlock+0x75/0x190 [ 6784.147021] ? _raw_spin_unlock+0x12/0x30 [ 6784.147028] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6784.147065] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6784.147098] ll_atomic_open+0x36a/0xd70 [lustre] [ 6784.147357] lookup_open+0x356/0xb40 [ 6784.147367] do_last+0x313/0xfc0 [ 6784.147372] ? nd_jump_root+0xe5/0x160 [ 6784.147377] ? path_init+0x437/0x520 [ 6784.147383] path_openat+0xf7/0x500 [ 6784.147388] ? do_raw_spin_unlock+0x75/0x190 [ 6784.147392] do_filp_open+0x99/0x140 [ 6784.147399] ? getname_flags+0x6e/0x330 [ 6784.147404] ? __check_object_size+0xff/0x256 [ 6784.147411] ? do_raw_spin_unlock+0x75/0x190 [ 6784.147415] ? _raw_spin_unlock+0x12/0x30 [ 6784.147420] do_sys_openat2+0x2b4/0x410 [ 6784.147424] do_sys_open+0x73/0xa0 [ 6784.147429] __x64_sys_openat+0x24/0x30 [ 6784.147432] do_syscall_64+0xc1/0x440 [ 6784.147438] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6784.147444] RIP: 0033:0x7fdc0afbf332 [ 6784.147452] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6784.147455] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6784.147465] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6784.147468] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6784.147470] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6784.147476] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6784.147478] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6784.664176] ? watchdog+0x40/0x40 [ 6784.665626] ? hrtimer_interrupt+0x128/0x2c0 [ 6784.667505] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6784.669759] ? apic_timer_interrupt+0xf/0x20 [ 6784.671542] [ 6784.672457] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6784.675515] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6784.678140] ? dump_stack+0xf6/0x10e [ 6784.679622] do_raw_spin_lock+0x177/0x190 [ 6784.681365] _raw_spin_lock+0x21/0x30 [ 6784.682949] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6784.685747] obd_mod_free+0x46/0x140 [ptlrpc] [ 6784.688485] mdc_commit_open+0x91/0xb0 [mdc] [ 6784.690415] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6784.692728] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6784.695014] after_reply+0xe2c/0x1780 [ptlrpc] [ 6784.697249] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6784.699373] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6784.701177] ? do_wait_intr+0xf0/0xf0 [ 6784.702431] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6784.705026] kthread+0x1d1/0x200 [ 6784.706631] ? set_kthread_struct+0x70/0x70 [ 6784.708445] ret_from_fork+0x1f/0x30 [ 6787.551599] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4762s! [ 6787.562784] Showing busy workqueues and worker pools: [ 6787.564598] workqueue events: flags=0x0 [ 6787.566709] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6787.571638] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6787.575098] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6787.579783] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6787.582607] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6787.586035] pending: kfree_rcu_monitor [ 6787.591778] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6787.598832] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6787.611921] workqueue events_long: flags=0x0 [ 6787.615135] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6787.619459] pending: lu_objects_flush [obdclass] [ 6787.621857] workqueue events_unbound: flags=0x2 [ 6787.623909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6787.626780] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6787.631765] workqueue events_power_efficient: flags=0x80 [ 6787.634050] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6787.637291] pending: neigh_periodic_work, check_lifetime [ 6787.644013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6787.652343] pending: do_cache_clean, neigh_periodic_work [ 6787.659729] workqueue mm_percpu_wq: flags=0x8 [ 6787.661917] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6787.664560] pending: lru_add_drain_per_cpu [ 6787.667803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6787.671126] pending: vmstat_update, lru_add_drain_per_cpu [ 6787.673212] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6787.676013] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6787.680449] workqueue writeback: flags=0x4a [ 6787.682373] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6787.684888] pending: wb_workfn, wb_workfn [ 6787.687044] workqueue xprtiod: flags=0xa [ 6787.688681] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6787.691786] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6787.694547] workqueue ipv6_addrconf: flags=0x40008 [ 6787.696673] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6787.698849] pending: addrconf_verify_work [ 6787.700177] workqueue knbd0-recv: flags=0x1a [ 6787.701766] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6787.705957] in-flight: 655:recv_work [ 6787.708406] workqueue ptlrpc_pinger: flags=0x2 [ 6787.709842] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6787.712309] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6787.715142] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4762s workers=4 idle: 36396 38 11188 [ 6787.719037] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6778s workers=2 idle: 658 [ 6788.051098] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6788.056275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6788.100764] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6788.113842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6788.120822] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6788.124173] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6788.130820] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6788.141673] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6788.149396] RAX: 000000000000788b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6788.154966] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6788.163091] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6788.165950] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6788.176266] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6788.188367] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6788.200358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6788.206632] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6788.212893] Call Trace: [ 6788.213819] [ 6788.214618] ? show_regs.cold.9+0x22/0x2f [ 6788.224121] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6788.231002] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6788.232667] ? watchdog+0x40/0x40 [ 6788.233895] ? hrtimer_interrupt+0x128/0x2c0 [ 6788.241158] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6788.246784] ? apic_timer_interrupt+0xf/0x20 [ 6788.257149] [ 6788.258065] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6788.266545] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6788.270069] do_raw_spin_lock+0x177/0x190 [ 6788.272892] _raw_spin_lock+0x21/0x30 [ 6788.275466] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6788.279253] process_one_work+0x2c8/0x700 [ 6788.280863] worker_thread+0x296/0x6e0 [ 6788.284803] ? rescuer_thread+0x570/0x570 [ 6788.287983] kthread+0x1d1/0x200 [ 6788.289772] ? set_kthread_struct+0x70/0x70 [ 6788.297874] ret_from_fork+0x1f/0x30 [ 6800.055324] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 6800.062187] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6800.114047] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6800.133768] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6800.147922] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6800.157154] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6800.169897] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6800.172741] RAX: 0000000000003c98 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6800.175601] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6800.178237] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6800.185567] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6800.192935] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6800.199613] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6800.208326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6800.212260] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6800.218964] Call Trace: [ 6800.220108] [ 6800.221347] ? show_regs.cold.9+0x22/0x2f [ 6800.227030] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6800.230943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6800.233956] ? watchdog+0x40/0x40 [ 6800.235037] ? hrtimer_interrupt+0x128/0x2c0 [ 6800.240173] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6800.242053] ? apic_timer_interrupt+0xf/0x20 [ 6800.244734] [ 6800.245907] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6800.248055] do_raw_spin_lock+0x177/0x190 [ 6800.249810] _raw_spin_lock+0x21/0x30 [ 6800.250979] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6800.255805] ? do_raw_spin_unlock+0x75/0x190 [ 6800.257984] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6800.261650] ? do_wait_intr+0xf0/0xf0 [ 6800.264266] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6800.266265] mdc_close+0x53b/0xd70 [mdc] [ 6800.268235] lmv_close+0x18d/0x490 [lmv] [ 6800.270975] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6800.273530] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6800.276370] ll_md_close+0x274/0x800 [lustre] [ 6800.277912] ll_file_release+0x128/0xc50 [lustre] [ 6800.281708] __fput+0x10d/0x440 [ 6800.282728] ____fput+0x12/0x20 [ 6800.284674] task_work_run+0xc6/0x110 [ 6800.288225] exit_to_usermode_loop+0x1dd/0x1f0 [ 6800.290610] do_syscall_64+0x3d6/0x440 [ 6800.293357] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6800.295721] RIP: 0033:0x7fdc0afbfcd8 [ 6800.298217] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6800.307712] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6800.312858] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6800.317944] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6800.321308] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6800.324895] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6800.328161] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6812.048258] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 6812.052322] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 6812.052599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6812.052682] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6812.052691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6812.052697] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6812.052714] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6812.052722] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6812.052730] RAX: 0000000000006289 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6812.052732] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6812.052735] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6812.052737] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6812.052739] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6812.052743] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6812.052745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6812.052752] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6812.052764] Call Trace: [ 6812.052774] [ 6812.052781] ? show_regs.cold.9+0x22/0x2f [ 6812.052795] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6812.052805] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6812.052814] ? watchdog+0x40/0x40 [ 6812.052819] ? hrtimer_interrupt+0x128/0x2c0 [ 6812.052827] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6812.052836] ? apic_timer_interrupt+0xf/0x20 [ 6812.052849] [ 6812.052852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6812.052857] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6812.052861] ? dump_stack+0xf6/0x10e [ 6812.052866] do_raw_spin_lock+0x177/0x190 [ 6812.052876] _raw_spin_lock+0x21/0x30 [ 6812.052883] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6812.053216] obd_mod_free+0x46/0x140 [ptlrpc] [ 6812.053481] mdc_commit_open+0x91/0xb0 [mdc] [ 6812.053560] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6812.053893] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6812.054181] after_reply+0xe2c/0x1780 [ptlrpc] [ 6812.054424] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6812.054942] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6812.055207] ? do_wait_intr+0xf0/0xf0 [ 6812.055216] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6812.055710] kthread+0x1d1/0x200 [ 6812.055717] ? set_kthread_struct+0x70/0x70 [ 6812.055721] ret_from_fork+0x1f/0x30 [ 6812.376620] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6812.414708] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6812.425293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6812.430371] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6812.437306] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6812.449294] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6812.452374] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6812.462018] RDX: 00000000000070a3 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6812.466122] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6812.469459] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6812.483549] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6812.488749] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6812.496325] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6812.502162] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6812.510026] Call Trace: [ 6812.514097] [ 6812.517377] ? show_regs.cold.9+0x22/0x2f [ 6812.520804] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6812.525863] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6812.530782] ? watchdog+0x40/0x40 [ 6812.531992] ? hrtimer_interrupt+0x128/0x2c0 [ 6812.536296] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6812.538597] ? apic_timer_interrupt+0xf/0x20 [ 6812.543538] [ 6812.546478] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6812.553550] do_raw_spin_lock+0x177/0x190 [ 6812.559068] _raw_spin_lock+0x21/0x30 [ 6812.560502] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6812.567358] ? do_raw_spin_unlock+0x75/0x190 [ 6812.571390] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6812.577325] ? do_wait_intr+0xf0/0xf0 [ 6812.579602] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6812.587452] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6812.593576] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6812.599744] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6812.607716] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6812.613845] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6812.616689] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6812.622621] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6812.628787] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6812.634976] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6812.645392] lmv_intent_open+0x338/0x1150 [lmv] [ 6812.649124] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6812.654301] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6812.658523] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6812.663924] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6812.665284] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6812.674229] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6812.679320] ? do_raw_spin_unlock+0x75/0x190 [ 6812.681110] ? _raw_spin_unlock+0x12/0x30 [ 6812.683686] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6812.687687] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6812.692436] ll_atomic_open+0x36a/0xd70 [lustre] [ 6812.694750] lookup_open+0x356/0xb40 [ 6812.700615] do_last+0x313/0xfc0 [ 6812.705384] ? nd_jump_root+0xe5/0x160 [ 6812.711400] ? path_init+0x437/0x520 [ 6812.715102] path_openat+0xf7/0x500 [ 6812.718470] ? do_raw_spin_unlock+0x75/0x190 [ 6812.720723] do_filp_open+0x99/0x140 [ 6812.724629] ? getname_flags+0x6e/0x330 [ 6812.729128] ? __check_object_size+0xff/0x256 [ 6812.730715] ? do_raw_spin_unlock+0x75/0x190 [ 6812.737623] ? _raw_spin_unlock+0x12/0x30 [ 6812.743673] do_sys_openat2+0x2b4/0x410 [ 6812.748265] do_sys_open+0x73/0xa0 [ 6812.753647] __x64_sys_openat+0x24/0x30 [ 6812.758702] do_syscall_64+0xc1/0x440 [ 6812.764617] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6812.766534] RIP: 0033:0x7fdc0afbf332 [ 6812.769725] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6812.786439] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6812.796110] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6812.804971] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6812.814110] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6812.823508] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6812.828959] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6816.054534] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6816.066673] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6816.113236] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6816.126227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6816.138585] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6816.146580] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6816.150863] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6816.166875] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6816.178211] RAX: 0000000000005399 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6816.187741] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6816.200099] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6816.214653] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6816.224573] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6816.234824] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6816.248899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6816.258167] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6816.268771] Call Trace: [ 6816.270825] [ 6816.276414] ? show_regs.cold.9+0x22/0x2f [ 6816.283919] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6816.285831] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6816.292431] ? watchdog+0x40/0x40 [ 6816.295586] ? hrtimer_interrupt+0x128/0x2c0 [ 6816.299825] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6816.305632] ? apic_timer_interrupt+0xf/0x20 [ 6816.311610] [ 6816.314977] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6816.318750] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6816.321622] do_raw_spin_lock+0x177/0x190 [ 6816.324829] _raw_spin_lock+0x21/0x30 [ 6816.327025] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6816.331622] process_one_work+0x2c8/0x700 [ 6816.336036] worker_thread+0x296/0x6e0 [ 6816.339533] ? rescuer_thread+0x570/0x570 [ 6816.344919] kthread+0x1d1/0x200 [ 6816.346175] ? set_kthread_struct+0x70/0x70 [ 6816.349796] ret_from_fork+0x1f/0x30 [ 6817.761637] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4792s! [ 6817.773382] Showing busy workqueues and worker pools: [ 6817.781982] workqueue events: flags=0x0 [ 6817.789814] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6817.808319] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6817.816688] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6817.828302] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6817.848984] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6817.851557] pending: kfree_rcu_monitor [ 6817.861173] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6817.863383] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6817.884682] workqueue events_long: flags=0x0 [ 6817.887120] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6817.894992] pending: lu_objects_flush [obdclass] [ 6817.900945] workqueue events_unbound: flags=0x2 [ 6817.903034] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6817.906015] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6817.923302] workqueue events_power_efficient: flags=0x80 [ 6817.929186] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6817.937842] pending: neigh_periodic_work, check_lifetime [ 6817.944734] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6817.963993] pending: do_cache_clean, neigh_periodic_work [ 6817.973672] workqueue mm_percpu_wq: flags=0x8 [ 6817.975145] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6817.991661] pending: lru_add_drain_per_cpu [ 6817.998780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6818.010759] pending: vmstat_update, lru_add_drain_per_cpu [ 6818.021473] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6818.023824] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6818.031539] workqueue writeback: flags=0x4a [ 6818.036362] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6818.042587] pending: wb_workfn, wb_workfn [ 6818.047018] workqueue xprtiod: flags=0xa [ 6818.049870] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6818.053578] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6818.065334] workqueue ipv6_addrconf: flags=0x40008 [ 6818.071047] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6818.076998] pending: addrconf_verify_work [ 6818.078316] workqueue knbd0-recv: flags=0x1a [ 6818.084662] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6818.088296] in-flight: 655:recv_work [ 6818.092726] workqueue ptlrpc_pinger: flags=0x2 [ 6818.095409] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6818.102951] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6818.110451] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4793s workers=4 idle: 36396 38 11188 [ 6818.117814] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6808s workers=2 idle: 658 [ 6828.056712] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 6828.061091] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6828.090295] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6828.100148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6828.107822] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6828.109114] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6828.124688] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6828.129729] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6828.132840] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6828.135708] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6828.138971] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6828.142273] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6828.145306] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6828.148834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6828.151881] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6828.156220] Call Trace: [ 6828.157740] [ 6828.158325] ? show_regs.cold.9+0x22/0x2f [ 6828.162803] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6828.166670] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6828.168103] ? watchdog+0x40/0x40 [ 6828.169685] ? hrtimer_interrupt+0x128/0x2c0 [ 6828.171450] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6828.176050] ? apic_timer_interrupt+0xf/0x20 [ 6828.180174] [ 6828.183112] ? native_safe_halt+0x2e/0x50 [ 6828.185003] kvm_wait+0xcd/0xe0 [ 6828.187627] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6828.192042] do_raw_spin_lock+0x177/0x190 [ 6828.194688] _raw_spin_lock+0x21/0x30 [ 6828.198683] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6828.205627] ? do_raw_spin_unlock+0x75/0x190 [ 6828.208136] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6828.211413] ? do_wait_intr+0xf0/0xf0 [ 6828.215446] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6828.217766] mdc_close+0x53b/0xd70 [mdc] [ 6828.219569] lmv_close+0x18d/0x490 [lmv] [ 6828.220962] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6828.223011] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6828.224854] ll_md_close+0x274/0x800 [lustre] [ 6828.226398] ll_file_release+0x128/0xc50 [lustre] [ 6828.228154] __fput+0x10d/0x440 [ 6828.229296] ____fput+0x12/0x20 [ 6828.230665] task_work_run+0xc6/0x110 [ 6828.232584] exit_to_usermode_loop+0x1dd/0x1f0 [ 6828.234735] do_syscall_64+0x3d6/0x440 [ 6828.243197] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6828.246730] RIP: 0033:0x7fdc0afbfcd8 [ 6828.249882] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6828.268820] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6828.274617] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6828.278241] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6828.284985] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6828.288797] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6828.293284] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6840.052764] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 6840.060679] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6840.115207] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6840.131787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6840.144244] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6840.147329] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6840.169339] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6840.178447] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6840.183264] RDX: 00000000000074ae RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6840.185854] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6840.197018] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6840.203859] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6840.208431] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6840.212869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6840.215337] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6840.226933] Call Trace: [ 6840.227836] [ 6840.234464] ? show_regs.cold.9+0x22/0x2f [ 6840.238475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6840.243182] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6840.248331] ? watchdog+0x40/0x40 [ 6840.249647] ? hrtimer_interrupt+0x128/0x2c0 [ 6840.253341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6840.255684] ? apic_timer_interrupt+0xf/0x20 [ 6840.259610] [ 6840.260522] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6840.264552] do_raw_spin_lock+0x177/0x190 [ 6840.267338] _raw_spin_lock+0x21/0x30 [ 6840.275380] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6840.280609] ? do_raw_spin_unlock+0x75/0x190 [ 6840.285643] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6840.294320] ? do_wait_intr+0xf0/0xf0 [ 6840.300137] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6840.302806] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6840.309837] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6840.312415] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6840.315045] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6840.316712] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6840.318904] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6840.320596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6840.322960] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6840.326387] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6840.328570] lmv_intent_open+0x338/0x1150 [lmv] [ 6840.330489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6840.332244] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6840.335436] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6840.337159] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6840.338554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6840.348402] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6840.353092] ? do_raw_spin_unlock+0x75/0x190 [ 6840.354878] ? _raw_spin_unlock+0x12/0x30 [ 6840.356407] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6840.360172] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6840.362766] ll_atomic_open+0x36a/0xd70 [lustre] [ 6840.365315] lookup_open+0x356/0xb40 [ 6840.368683] do_last+0x313/0xfc0 [ 6840.370059] ? nd_jump_root+0xe5/0x160 [ 6840.371618] ? path_init+0x437/0x520 [ 6840.373184] path_openat+0xf7/0x500 [ 6840.375444] ? do_raw_spin_unlock+0x75/0x190 [ 6840.377403] do_filp_open+0x99/0x140 [ 6840.379381] ? getname_flags+0x6e/0x330 [ 6840.381340] ? __check_object_size+0xff/0x256 [ 6840.384650] ? do_raw_spin_unlock+0x75/0x190 [ 6840.387070] ? _raw_spin_unlock+0x12/0x30 [ 6840.393779] do_sys_openat2+0x2b4/0x410 [ 6840.395265] do_sys_open+0x73/0xa0 [ 6840.399124] __x64_sys_openat+0x24/0x30 [ 6840.400890] do_syscall_64+0xc1/0x440 [ 6840.402960] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6840.404941] RIP: 0033:0x7fdc0afbf332 [ 6840.408043] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6840.418215] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6840.422646] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6840.425056] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6840.427395] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6840.430363] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6840.433661] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6844.048848] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 6844.051012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6844.062075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [ 6844.073092] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6844.073174] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6844.073182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6844.073189] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6844.073484] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6844.073495] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6844.073499] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6844.073504] RAX: 000000000000005b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6844.073506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6844.073509] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6844.073511] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6844.073513] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6844.073517] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6844.073519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6844.073521] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6844.073532] Call Trace: [ 6844.073537] [ 6844.073543] ? show_regs.cold.9+0x22/0x2f [ 6844.073551] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6844.073557] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6844.073561] ? watchdog+0x40/0x40 [ 6844.073566] ? hrtimer_interrupt+0x128/0x2c0 [ 6844.074974] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6844.074983] ? apic_timer_interrupt+0xf/0x20 [ 6844.074988] [ 6844.074992] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6844.074998] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6844.075005] do_raw_spin_lock+0x177/0x190 [ 6844.075011] _raw_spin_lock+0x21/0x30 [ 6844.075015] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6844.075296] process_one_work+0x2c8/0x700 [ 6844.075304] worker_thread+0x296/0x6e0 [ 6844.075309] ? rescuer_thread+0x570/0x570 [ 6844.075312] kthread+0x1d1/0x200 [ 6844.075317] ? set_kthread_struct+0x70/0x70 [ 6844.075321] ret_from_fork+0x1f/0x30 [ 6844.201964] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6844.218053] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6844.223214] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6844.227733] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6844.230533] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6844.239866] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6844.243600] RAX: 0000000000004c89 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6844.247321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6844.250079] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6844.253454] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6844.256582] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6844.265732] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6844.273265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6844.276214] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6844.285142] Call Trace: [ 6844.285993] [ 6844.286736] ? show_regs.cold.9+0x22/0x2f [ 6844.293200] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6844.298457] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6844.300437] ? watchdog+0x40/0x40 [ 6844.301625] ? hrtimer_interrupt+0x128/0x2c0 [ 6844.304901] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6844.309476] ? apic_timer_interrupt+0xf/0x20 [ 6844.314599] [ 6844.315909] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6844.319971] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6844.325257] ? dump_stack+0xf6/0x10e [ 6844.333863] do_raw_spin_lock+0x177/0x190 [ 6844.340693] _raw_spin_lock+0x21/0x30 [ 6844.344515] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6844.351139] obd_mod_free+0x46/0x140 [ptlrpc] [ 6844.360110] mdc_commit_open+0x91/0xb0 [mdc] [ 6844.362259] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6844.367505] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6844.376140] after_reply+0xe2c/0x1780 [ptlrpc] [ 6844.380442] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6844.385265] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6844.390608] ? do_wait_intr+0xf0/0xf0 [ 6844.394460] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6844.403162] kthread+0x1d1/0x200 [ 6844.405145] ? set_kthread_struct+0x70/0x70 [ 6844.406254] ret_from_fork+0x1f/0x30 [ 6848.479442] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4823s! [ 6848.484542] Showing busy workqueues and worker pools: [ 6848.488494] workqueue events: flags=0x0 [ 6848.492228] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6848.501302] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6848.506449] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6848.514411] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6848.524345] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6848.529949] pending: kfree_rcu_monitor [ 6848.534051] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6848.541480] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6848.551315] workqueue events_long: flags=0x0 [ 6848.555126] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6848.559401] pending: lu_objects_flush [obdclass] [ 6848.567087] workqueue events_unbound: flags=0x2 [ 6848.574017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6848.583156] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6848.596229] workqueue events_power_efficient: flags=0x80 [ 6848.603983] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6848.608222] pending: neigh_periodic_work, check_lifetime [ 6848.612878] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6848.622305] pending: do_cache_clean, neigh_periodic_work [ 6848.630123] workqueue mm_percpu_wq: flags=0x8 [ 6848.637254] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6848.644028] pending: lru_add_drain_per_cpu [ 6848.648474] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6848.663400] pending: vmstat_update, lru_add_drain_per_cpu [ 6848.665044] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6848.677299] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6848.687932] workqueue writeback: flags=0x4a [ 6848.693808] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6848.706987] pending: wb_workfn, wb_workfn [ 6848.711529] workqueue xprtiod: flags=0xa [ 6848.712768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6848.723238] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6848.739052] workqueue ipv6_addrconf: flags=0x40008 [ 6848.746228] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6848.752176] pending: addrconf_verify_work [ 6848.753484] workqueue knbd0-recv: flags=0x1a [ 6848.763110] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6848.768317] in-flight: 655:recv_work [ 6848.774408] workqueue ptlrpc_pinger: flags=0x2 [ 6848.778461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6848.783030] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6848.794152] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4823s workers=4 idle: 36396 38 11188 [ 6848.801735] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6839s workers=2 idle: 658 [ 6856.054081] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 6856.056662] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6856.088276] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6856.097616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6856.108525] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6856.110248] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6856.124707] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6856.130082] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6856.133012] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6856.142265] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6856.147382] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6856.153442] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6856.161351] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6856.164961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6856.167008] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6856.169586] Call Trace: [ 6856.170384] [ 6856.171103] ? show_regs.cold.9+0x22/0x2f [ 6856.172991] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6856.176216] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6856.182679] ? watchdog+0x40/0x40 [ 6856.184169] ? hrtimer_interrupt+0x128/0x2c0 [ 6856.185732] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6856.188320] ? apic_timer_interrupt+0xf/0x20 [ 6856.192998] [ 6856.194709] ? native_safe_halt+0x2e/0x50 [ 6856.197240] kvm_wait+0xcd/0xe0 [ 6856.198568] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6856.200347] do_raw_spin_lock+0x177/0x190 [ 6856.201723] _raw_spin_lock+0x21/0x30 [ 6856.202979] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6856.204930] ? do_raw_spin_unlock+0x75/0x190 [ 6856.206159] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6856.207844] ? do_wait_intr+0xf0/0xf0 [ 6856.208917] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6856.210694] mdc_close+0x53b/0xd70 [mdc] [ 6856.212088] lmv_close+0x18d/0x490 [lmv] [ 6856.213297] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6856.215012] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6856.216534] ll_md_close+0x274/0x800 [lustre] [ 6856.220526] ll_file_release+0x128/0xc50 [lustre] [ 6856.221903] __fput+0x10d/0x440 [ 6856.225887] ____fput+0x12/0x20 [ 6856.226930] task_work_run+0xc6/0x110 [ 6856.228018] exit_to_usermode_loop+0x1dd/0x1f0 [ 6856.240068] do_syscall_64+0x3d6/0x440 [ 6856.244728] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6856.247068] RIP: 0033:0x7fdc0afbfcd8 [ 6856.249067] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6856.261662] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6856.266235] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6856.271396] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6856.275285] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6856.280298] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6856.287740] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6868.053291] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6868.058293] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6868.090505] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6868.095966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6868.104571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6868.110999] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6868.120276] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6868.122971] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6868.125719] RDX: 000000000000390b RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6868.128597] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6868.131422] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6868.133765] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6868.136367] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6868.139181] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6868.140829] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6868.143412] Call Trace: [ 6868.144190] [ 6868.145014] ? show_regs.cold.9+0x22/0x2f [ 6868.146764] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6868.148642] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6868.150137] ? watchdog+0x40/0x40 [ 6868.151290] ? hrtimer_interrupt+0x128/0x2c0 [ 6868.152493] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6868.154266] ? apic_timer_interrupt+0xf/0x20 [ 6868.156398] [ 6868.157119] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6868.160414] do_raw_spin_lock+0x177/0x190 [ 6868.165408] _raw_spin_lock+0x21/0x30 [ 6868.167486] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6868.171921] ? do_raw_spin_unlock+0x75/0x190 [ 6868.173443] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6868.178040] ? do_wait_intr+0xf0/0xf0 [ 6868.181732] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6868.185745] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6868.190959] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6868.193996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6868.199881] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6868.210962] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6868.215614] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6868.218798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6868.220895] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6868.227530] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6868.233666] lmv_intent_open+0x338/0x1150 [lmv] [ 6868.236587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6868.240906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6868.242655] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6868.247706] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6868.249723] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6868.251954] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6868.256245] ? do_raw_spin_unlock+0x75/0x190 [ 6868.258080] ? _raw_spin_unlock+0x12/0x30 [ 6868.261342] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6868.265013] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6868.267257] ll_atomic_open+0x36a/0xd70 [lustre] [ 6868.269578] lookup_open+0x356/0xb40 [ 6868.272896] do_last+0x313/0xfc0 [ 6868.275390] ? nd_jump_root+0xe5/0x160 [ 6868.278438] ? path_init+0x437/0x520 [ 6868.280560] path_openat+0xf7/0x500 [ 6868.283784] ? do_raw_spin_unlock+0x75/0x190 [ 6868.285472] do_filp_open+0x99/0x140 [ 6868.286975] ? getname_flags+0x6e/0x330 [ 6868.290543] ? __check_object_size+0xff/0x256 [ 6868.291899] ? do_raw_spin_unlock+0x75/0x190 [ 6868.294417] ? _raw_spin_unlock+0x12/0x30 [ 6868.298570] do_sys_openat2+0x2b4/0x410 [ 6868.304194] do_sys_open+0x73/0xa0 [ 6868.305395] __x64_sys_openat+0x24/0x30 [ 6868.310799] do_syscall_64+0xc1/0x440 [ 6868.312337] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6868.316327] RIP: 0033:0x7fdc0afbf332 [ 6868.317521] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6868.328951] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6868.339884] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6868.344896] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6868.350444] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6868.355518] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6868.357461] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6872.051028] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6872.053692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6872.101431] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6872.114411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6872.122743] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6872.130941] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6872.133573] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6872.154716] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6872.158990] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6872.165105] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6872.176249] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6872.178815] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6872.183610] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6872.192043] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6872.199191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6872.203676] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6872.208024] Call Trace: [ 6872.209582] [ 6872.211022] ? show_regs.cold.9+0x22/0x2f [ 6872.213930] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6872.218173] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6872.223714] ? watchdog+0x40/0x40 [ 6872.228563] ? hrtimer_interrupt+0x128/0x2c0 [ 6872.233993] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6872.238995] ? apic_timer_interrupt+0xf/0x20 [ 6872.241612] [ 6872.242983] ? native_safe_halt+0x2e/0x50 [ 6872.245977] kvm_wait+0xcd/0xe0 [ 6872.248002] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6872.253089] do_raw_spin_lock+0x177/0x190 [ 6872.259022] _raw_spin_lock+0x21/0x30 [ 6872.263631] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6872.270526] process_one_work+0x2c8/0x700 [ 6872.273100] worker_thread+0x296/0x6e0 [ 6872.276048] ? rescuer_thread+0x570/0x570 [ 6872.281781] kthread+0x1d1/0x200 [ 6872.283859] ? set_kthread_struct+0x70/0x70 [ 6872.289930] ret_from_fork+0x1f/0x30 [ 6876.048239] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6876.054245] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6876.097520] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6876.107980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6876.118834] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6876.124002] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 6876.145647] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6876.154755] RAX: 00000000000054ed RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6876.159506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6876.169400] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6876.182232] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6876.184834] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6876.188419] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6876.192267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6876.194806] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6876.198009] Call Trace: [ 6876.199111] [ 6876.199824] ? show_regs.cold.9+0x22/0x2f [ 6876.201721] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6876.203553] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6876.205368] ? watchdog+0x40/0x40 [ 6876.206577] ? hrtimer_interrupt+0x128/0x2c0 [ 6876.208646] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6876.211030] ? apic_timer_interrupt+0xf/0x20 [ 6876.213125] [ 6876.213955] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6876.216613] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6876.218841] ? dump_stack+0xf6/0x10e [ 6876.220413] do_raw_spin_lock+0x177/0x190 [ 6876.221791] _raw_spin_lock+0x21/0x30 [ 6876.223494] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6876.225881] obd_mod_free+0x46/0x140 [ptlrpc] [ 6876.228536] mdc_commit_open+0x91/0xb0 [mdc] [ 6876.230620] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6876.232841] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6876.235566] after_reply+0xe2c/0x1780 [ptlrpc] [ 6876.238025] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6876.240856] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6876.242932] ? do_wait_intr+0xf0/0xf0 [ 6876.245280] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6876.248818] kthread+0x1d1/0x200 [ 6876.249969] ? set_kthread_struct+0x70/0x70 [ 6876.251506] ret_from_fork+0x1f/0x30 [ 6879.199715] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4854s! [ 6879.203880] Showing busy workqueues and worker pools: [ 6879.205671] workqueue events: flags=0x0 [ 6879.207474] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6879.211147] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6879.214595] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6879.217612] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6879.224366] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6879.226682] pending: kfree_rcu_monitor [ 6879.228185] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6879.231209] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6879.236729] workqueue events_long: flags=0x0 [ 6879.238794] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6879.241352] pending: lu_objects_flush [obdclass] [ 6879.243566] workqueue events_unbound: flags=0x2 [ 6879.245069] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6879.247236] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6879.251692] workqueue events_power_efficient: flags=0x80 [ 6879.253792] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6879.256425] pending: neigh_periodic_work, check_lifetime [ 6879.258531] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6879.260253] pending: do_cache_clean, neigh_periodic_work [ 6879.262305] workqueue mm_percpu_wq: flags=0x8 [ 6879.263854] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6879.266546] pending: lru_add_drain_per_cpu [ 6879.268325] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6879.270993] pending: vmstat_update, lru_add_drain_per_cpu [ 6879.275232] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6879.278230] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6879.280680] workqueue writeback: flags=0x4a [ 6879.282401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6879.285319] pending: wb_workfn, wb_workfn [ 6879.287448] workqueue xprtiod: flags=0xa [ 6879.290540] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6879.293362] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6879.299986] workqueue ipv6_addrconf: flags=0x40008 [ 6879.303113] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6879.309075] pending: addrconf_verify_work [ 6879.310744] workqueue knbd0-recv: flags=0x1a [ 6879.312867] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6879.315262] in-flight: 655:recv_work [ 6879.316937] workqueue ptlrpc_pinger: flags=0x2 [ 6879.318757] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6879.321000] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6879.323234] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4854s workers=4 idle: 36396 38 11188 [ 6879.327174] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6869s workers=2 idle: 658 [ 6884.056539] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 6884.065661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6884.103152] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6884.109124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6884.114091] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6884.117796] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6884.130959] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6884.137811] RAX: 0000000000007289 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6884.142587] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6884.146676] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6884.151069] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6884.155373] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6884.158746] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6884.163446] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6884.167018] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6884.170744] Call Trace: [ 6884.172082] [ 6884.173369] ? show_regs.cold.9+0x22/0x2f [ 6884.175066] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6884.177693] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6884.180080] ? watchdog+0x40/0x40 [ 6884.182381] ? hrtimer_interrupt+0x128/0x2c0 [ 6884.184684] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6884.187107] ? apic_timer_interrupt+0xf/0x20 [ 6884.189553] [ 6884.190313] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6884.192743] do_raw_spin_lock+0x177/0x190 [ 6884.194819] _raw_spin_lock+0x21/0x30 [ 6884.197491] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6884.200318] ? do_raw_spin_unlock+0x75/0x190 [ 6884.202709] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6884.206075] ? do_wait_intr+0xf0/0xf0 [ 6884.208480] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6884.211491] mdc_close+0x53b/0xd70 [mdc] [ 6884.213019] lmv_close+0x18d/0x490 [lmv] [ 6884.215493] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6884.218446] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6884.222198] ll_md_close+0x274/0x800 [lustre] [ 6884.224814] ll_file_release+0x128/0xc50 [lustre] [ 6884.227948] __fput+0x10d/0x440 [ 6884.229683] ____fput+0x12/0x20 [ 6884.231691] task_work_run+0xc6/0x110 [ 6884.233662] exit_to_usermode_loop+0x1dd/0x1f0 [ 6884.236013] do_syscall_64+0x3d6/0x440 [ 6884.238759] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6884.242545] RIP: 0033:0x7fdc0afbfcd8 [ 6884.244672] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6884.255021] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6884.259199] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6884.262614] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6884.266389] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6884.269492] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6884.272986] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6896.052015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6896.056485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6896.085702] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6896.092027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6896.096539] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6896.100044] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6896.118107] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6896.124034] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6896.126684] RDX: 00000000000009bd RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6896.135151] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6896.145893] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6896.153318] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6896.162220] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6896.175705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6896.182358] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6896.191654] Call Trace: [ 6896.194904] [ 6896.200567] ? show_regs.cold.9+0x22/0x2f [ 6896.207405] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6896.211572] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6896.214934] ? watchdog+0x40/0x40 [ 6896.220434] ? hrtimer_interrupt+0x128/0x2c0 [ 6896.228125] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6896.233436] ? apic_timer_interrupt+0xf/0x20 [ 6896.239814] [ 6896.244909] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6896.252100] do_raw_spin_lock+0x177/0x190 [ 6896.259729] _raw_spin_lock+0x21/0x30 [ 6896.264842] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6896.276419] ? do_raw_spin_unlock+0x75/0x190 [ 6896.284253] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6896.288799] ? do_wait_intr+0xf0/0xf0 [ 6896.294099] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6896.299630] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6896.310369] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6896.316852] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6896.323856] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6896.329848] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6896.334327] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6896.339224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6896.351769] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6896.359641] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6896.368754] lmv_intent_open+0x338/0x1150 [lmv] [ 6896.376787] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6896.386170] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6896.390597] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6896.394822] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6896.405805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6896.412320] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6896.416396] ? do_raw_spin_unlock+0x75/0x190 [ 6896.425372] ? _raw_spin_unlock+0x12/0x30 [ 6896.426694] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6896.436955] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6896.442626] ll_atomic_open+0x36a/0xd70 [lustre] [ 6896.449337] lookup_open+0x356/0xb40 [ 6896.451676] do_last+0x313/0xfc0 [ 6896.459791] ? nd_jump_root+0xe5/0x160 [ 6896.461594] ? path_init+0x437/0x520 [ 6896.468342] path_openat+0xf7/0x500 [ 6896.471702] ? do_raw_spin_unlock+0x75/0x190 [ 6896.478848] do_filp_open+0x99/0x140 [ 6896.482304] ? getname_flags+0x6e/0x330 [ 6896.487296] ? __check_object_size+0xff/0x256 [ 6896.493344] ? do_raw_spin_unlock+0x75/0x190 [ 6896.498749] ? _raw_spin_unlock+0x12/0x30 [ 6896.505034] do_sys_openat2+0x2b4/0x410 [ 6896.510421] do_sys_open+0x73/0xa0 [ 6896.516420] __x64_sys_openat+0x24/0x30 [ 6896.525367] do_syscall_64+0xc1/0x440 [ 6896.531863] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6896.538138] RIP: 0033:0x7fdc0afbf332 [ 6896.539535] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6896.575374] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6896.581985] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6896.589182] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6896.600945] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6896.609677] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6896.618654] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6900.052473] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6900.059274] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6900.134076] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6900.149718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6900.185335] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6900.206122] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6900.211354] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6900.255243] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6900.270370] RAX: 0000000000007789 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6900.285581] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6900.293794] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6900.306796] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6900.321760] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6900.327402] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6900.333790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6900.337267] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6900.349992] Call Trace: [ 6900.351550] [ 6900.353316] ? show_regs.cold.9+0x22/0x2f [ 6900.358932] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6900.367778] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6900.370880] ? watchdog+0x40/0x40 [ 6900.374574] ? hrtimer_interrupt+0x128/0x2c0 [ 6900.377997] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6900.383875] ? apic_timer_interrupt+0xf/0x20 [ 6900.387401] [ 6900.392283] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6900.400280] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6900.408817] do_raw_spin_lock+0x177/0x190 [ 6900.412492] _raw_spin_lock+0x21/0x30 [ 6900.416737] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6900.421419] process_one_work+0x2c8/0x700 [ 6900.424849] worker_thread+0x296/0x6e0 [ 6900.427722] ? rescuer_thread+0x570/0x570 [ 6900.433379] kthread+0x1d1/0x200 [ 6900.440102] ? set_kthread_struct+0x70/0x70 [ 6900.441644] ret_from_fork+0x1f/0x30 [ 6904.048021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6904.053039] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6904.079887] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6904.086504] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6904.091698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6904.095419] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6904.106950] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6904.111508] RAX: 0000000000003c05 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6904.116475] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6904.121015] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6904.125254] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6904.129814] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6904.133732] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6904.139294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6904.142486] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6904.146552] Call Trace: [ 6904.148182] [ 6904.149528] ? show_regs.cold.9+0x22/0x2f [ 6904.152171] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6904.154861] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6904.157553] ? watchdog+0x40/0x40 [ 6904.159587] ? hrtimer_interrupt+0x128/0x2c0 [ 6904.162853] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6904.165676] ? apic_timer_interrupt+0xf/0x20 [ 6904.167977] [ 6904.168752] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6904.171926] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6904.175288] ? dump_stack+0xf6/0x10e [ 6904.177443] do_raw_spin_lock+0x177/0x190 [ 6904.180166] _raw_spin_lock+0x21/0x30 [ 6904.182279] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6904.185962] obd_mod_free+0x46/0x140 [ptlrpc] [ 6904.189373] mdc_commit_open+0x91/0xb0 [mdc] [ 6904.192438] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6904.196199] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6904.199950] after_reply+0xe2c/0x1780 [ptlrpc] [ 6904.203214] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6904.207259] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6904.210625] ? do_wait_intr+0xf0/0xf0 [ 6904.212889] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6904.217008] kthread+0x1d1/0x200 [ 6904.219377] ? set_kthread_struct+0x70/0x70 [ 6904.221922] ret_from_fork+0x1f/0x30 [ 6909.407157] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4884s! [ 6909.411946] Showing busy workqueues and worker pools: [ 6909.424115] workqueue events: flags=0x0 [ 6909.425919] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6909.435129] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6909.445994] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6909.455201] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6909.463839] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6909.469946] pending: kfree_rcu_monitor [ 6909.479234] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6909.487774] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6909.506581] workqueue events_long: flags=0x0 [ 6909.509680] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6909.516148] pending: lu_objects_flush [obdclass] [ 6909.520019] workqueue events_unbound: flags=0x2 [ 6909.525204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6909.537943] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6909.555520] workqueue events_power_efficient: flags=0x80 [ 6909.559667] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6909.570065] pending: neigh_periodic_work, check_lifetime [ 6909.574143] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6909.578407] pending: do_cache_clean, neigh_periodic_work [ 6909.581925] workqueue mm_percpu_wq: flags=0x8 [ 6909.588383] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6909.590856] pending: lru_add_drain_per_cpu [ 6909.606010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6909.611189] pending: vmstat_update, lru_add_drain_per_cpu [ 6909.614915] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6909.626115] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6909.639224] workqueue writeback: flags=0x4a [ 6909.643406] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6909.652301] pending: wb_workfn, wb_workfn [ 6909.661738] workqueue xprtiod: flags=0xa [ 6909.670189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6909.677443] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6909.686327] workqueue ipv6_addrconf: flags=0x40008 [ 6909.692318] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6909.694557] pending: addrconf_verify_work [ 6909.704835] workqueue knbd0-recv: flags=0x1a [ 6909.708938] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6909.719703] in-flight: 655:recv_work [ 6909.726115] workqueue ptlrpc_pinger: flags=0x2 [ 6909.731849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6909.737253] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6909.740160] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4884s workers=4 idle: 36396 38 11188 [ 6909.744066] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6900s workers=2 idle: 658 [ 6912.054060] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6912.056594] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6912.073543] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6912.077982] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6912.081352] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6912.082896] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6912.090253] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6912.093315] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6912.096274] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6912.098573] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6912.101398] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6912.104159] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6912.106337] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6912.108726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6912.110587] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6912.113412] Call Trace: [ 6912.114375] [ 6912.115334] ? show_regs.cold.9+0x22/0x2f [ 6912.116504] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6912.118287] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6912.120177] ? watchdog+0x40/0x40 [ 6912.121581] ? hrtimer_interrupt+0x128/0x2c0 [ 6912.123488] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6912.125641] ? apic_timer_interrupt+0xf/0x20 [ 6912.127282] [ 6912.128485] ? native_safe_halt+0x2e/0x50 [ 6912.129979] kvm_wait+0xcd/0xe0 [ 6912.131329] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6912.133619] do_raw_spin_lock+0x177/0x190 [ 6912.135306] _raw_spin_lock+0x21/0x30 [ 6912.136743] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6912.139120] ? do_raw_spin_unlock+0x75/0x190 [ 6912.140584] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6912.144176] ? do_wait_intr+0xf0/0xf0 [ 6912.146585] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6912.148949] mdc_close+0x53b/0xd70 [mdc] [ 6912.150349] lmv_close+0x18d/0x490 [lmv] [ 6912.151804] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6912.155367] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6912.157271] ll_md_close+0x274/0x800 [lustre] [ 6912.159209] ll_file_release+0x128/0xc50 [lustre] [ 6912.161257] __fput+0x10d/0x440 [ 6912.163128] ____fput+0x12/0x20 [ 6912.165166] task_work_run+0xc6/0x110 [ 6912.166324] exit_to_usermode_loop+0x1dd/0x1f0 [ 6912.168241] do_syscall_64+0x3d6/0x440 [ 6912.169757] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6912.172123] RIP: 0033:0x7fdc0afbfcd8 [ 6912.174111] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6912.182414] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6912.185331] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6912.187711] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6912.190329] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6912.192823] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6912.195385] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6924.052049] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6924.055283] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6924.073126] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6924.077621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6924.080849] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6924.083138] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6924.089943] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6924.092790] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6924.095532] RDX: 0000000000000f3f RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6924.098387] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6924.100808] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6924.103157] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6924.105640] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6924.108922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6924.110935] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6924.113646] Call Trace: [ 6924.114807] [ 6924.115604] ? show_regs.cold.9+0x22/0x2f [ 6924.119121] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6924.121641] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6924.123493] ? watchdog+0x40/0x40 [ 6924.124868] ? hrtimer_interrupt+0x128/0x2c0 [ 6924.126189] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6924.127638] ? apic_timer_interrupt+0xf/0x20 [ 6924.128991] [ 6924.129831] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6924.132057] do_raw_spin_lock+0x177/0x190 [ 6924.133813] _raw_spin_lock+0x21/0x30 [ 6924.135281] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6924.139023] ? do_raw_spin_unlock+0x75/0x190 [ 6924.140841] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6924.143084] ? do_wait_intr+0xf0/0xf0 [ 6924.144836] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6924.153227] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6924.156164] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6924.159882] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6924.163063] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6924.165970] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6924.168167] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6924.171118] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6924.175021] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6924.178645] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6924.181957] lmv_intent_open+0x338/0x1150 [lmv] [ 6924.192759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6924.203090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6924.204984] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6924.216069] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6924.218601] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6924.224539] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6924.227695] ? do_raw_spin_unlock+0x75/0x190 [ 6924.233095] ? _raw_spin_unlock+0x12/0x30 [ 6924.236125] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6924.246147] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6924.247993] ll_atomic_open+0x36a/0xd70 [lustre] [ 6924.260287] lookup_open+0x356/0xb40 [ 6924.261682] do_last+0x313/0xfc0 [ 6924.273121] ? nd_jump_root+0xe5/0x160 [ 6924.275092] ? path_init+0x437/0x520 [ 6924.280522] path_openat+0xf7/0x500 [ 6924.281734] ? do_raw_spin_unlock+0x75/0x190 [ 6924.283229] do_filp_open+0x99/0x140 [ 6924.298044] ? getname_flags+0x6e/0x330 [ 6924.299321] ? __check_object_size+0xff/0x256 [ 6924.307548] ? do_raw_spin_unlock+0x75/0x190 [ 6924.311990] ? _raw_spin_unlock+0x12/0x30 [ 6924.315521] do_sys_openat2+0x2b4/0x410 [ 6924.319554] do_sys_open+0x73/0xa0 [ 6924.322033] __x64_sys_openat+0x24/0x30 [ 6924.324380] do_syscall_64+0xc1/0x440 [ 6924.328254] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6924.329984] RIP: 0033:0x7fdc0afbf332 [ 6924.335458] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6924.355820] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6924.362094] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6924.368935] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6924.379720] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6924.388814] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6924.396777] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6928.051028] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 6928.053674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6928.083493] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6928.089284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6928.093813] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6928.097434] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6928.099698] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6928.109918] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6928.114097] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6928.117500] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6928.121454] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6928.127460] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6928.133295] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6928.139280] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6928.143965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6928.147850] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6928.152417] Call Trace: [ 6928.154458] [ 6928.155515] ? show_regs.cold.9+0x22/0x2f [ 6928.158043] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6928.161041] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6928.164293] ? watchdog+0x40/0x40 [ 6928.166308] ? hrtimer_interrupt+0x128/0x2c0 [ 6928.168988] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6928.171498] ? apic_timer_interrupt+0xf/0x20 [ 6928.172914] [ 6928.174153] ? native_safe_halt+0x2e/0x50 [ 6928.176039] kvm_wait+0xcd/0xe0 [ 6928.177857] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6928.181199] do_raw_spin_lock+0x177/0x190 [ 6928.184104] _raw_spin_lock+0x21/0x30 [ 6928.186075] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6928.189868] process_one_work+0x2c8/0x700 [ 6928.192298] worker_thread+0x296/0x6e0 [ 6928.194855] ? rescuer_thread+0x570/0x570 [ 6928.197114] kthread+0x1d1/0x200 [ 6928.198728] ? set_kthread_struct+0x70/0x70 [ 6928.201098] ret_from_fork+0x1f/0x30 [ 6936.051509] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6936.059835] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6936.101751] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6936.115899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6936.123249] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6936.133223] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6936.156274] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6936.163143] RAX: 0000000000007f16 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6936.167361] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6936.171657] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6936.180293] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6936.187944] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6936.195955] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6936.205387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6936.208356] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6936.215332] Call Trace: [ 6936.220275] [ 6936.223430] ? show_regs.cold.9+0x22/0x2f [ 6936.227896] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6936.232699] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6936.234360] ? watchdog+0x40/0x40 [ 6936.241188] ? hrtimer_interrupt+0x128/0x2c0 [ 6936.244841] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6936.247329] ? apic_timer_interrupt+0xf/0x20 [ 6936.251874] [ 6936.254452] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6936.262543] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6936.264537] ? dump_stack+0xf6/0x10e [ 6936.267822] do_raw_spin_lock+0x177/0x190 [ 6936.273186] _raw_spin_lock+0x21/0x30 [ 6936.276902] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6936.282224] obd_mod_free+0x46/0x140 [ptlrpc] [ 6936.289254] mdc_commit_open+0x91/0xb0 [mdc] [ 6936.294741] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6936.300082] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6936.303327] after_reply+0xe2c/0x1780 [ptlrpc] [ 6936.309290] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6936.315572] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6936.319683] ? do_wait_intr+0xf0/0xf0 [ 6936.322428] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6936.324016] kthread+0x1d1/0x200 [ 6936.325087] ? set_kthread_struct+0x70/0x70 [ 6936.328769] ret_from_fork+0x1f/0x30 [ 6939.449307] rcu: INFO: rcu_sched self-detected stall on CPU [ 6939.453575] rcu: 0-...!: (2994987 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 6939.458279] (t=4920106 jiffies g=270761 q=3872) [ 6939.460089] rcu: rcu_sched kthread starved for 4915038 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6939.465397] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6939.469142] rcu: RCU grace-period kthread stack dump: [ 6939.471882] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6939.477535] Call Trace: [ 6939.478569] __schedule+0x351/0xcb0 [ 6939.480343] schedule+0xc0/0x180 [ 6939.481896] schedule_timeout+0xb4/0x190 [ 6939.483473] ? do_raw_spin_unlock+0x75/0x190 [ 6939.485413] ? __next_timer_interrupt+0x160/0x160 [ 6939.487597] ? prepare_to_swait_event+0xf3/0x140 [ 6939.488940] rcu_gp_kthread+0x8dd/0x15b0 [ 6939.490269] ? force_qs_rnp+0x2c0/0x2c0 [ 6939.492007] kthread+0x1d1/0x200 [ 6939.492988] ? set_kthread_struct+0x70/0x70 [ 6939.495319] ret_from_fork+0x1f/0x30 [ 6939.496919] rcu: Stack dump where RCU GP kthread last ran: [ 6939.498858] Sending NMI from CPU 0 to CPUs 1: [ 6939.500519] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 6939.502870] NMI backtrace for cpu 0 [ 6939.503937] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6939.508920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6939.512659] Call Trace: [ 6939.513556] [ 6939.514386] ? dump_stack+0xbb/0x10e [ 6939.515442] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6939.517189] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6939.519223] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6939.520915] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6939.522589] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6939.524429] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6939.526065] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6939.528238] ? update_process_times+0x6d/0xc0 [ 6939.529666] ? tick_sched_handle.isra.18+0x27/0x90 [ 6939.531609] ? tick_sched_timer+0x53/0xb0 [ 6939.533153] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6939.534608] ? tick_sched_handle.isra.18+0x90/0x90 [ 6939.536209] ? hrtimer_interrupt+0x128/0x2c0 [ 6939.537528] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6939.539227] ? apic_timer_interrupt+0xf/0x20 [ 6939.540799] [ 6939.541477] ? native_safe_halt+0x2e/0x50 [ 6939.542803] ? kvm_wait+0xcd/0xe0 [ 6939.544230] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6939.546298] ? dump_stack+0xf6/0x10e [ 6939.547521] ? do_raw_spin_lock+0x177/0x190 [ 6939.549085] ? _raw_spin_lock+0x21/0x30 [ 6939.550507] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6939.552499] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6939.554818] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6939.556541] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6939.559073] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6939.564168] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6939.568291] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6939.573330] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6939.576868] ? do_wait_intr+0xf0/0xf0 [ 6939.578571] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6939.581096] ? kthread+0x1d1/0x200 [ 6939.583533] ? set_kthread_struct+0x70/0x70 [ 6939.592505] ? ret_from_fork+0x1f/0x30 [ 6940.064059] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6940.073893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6940.127166] BUG: workqueue lockup - pool [ 6940.151424] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6940.151440] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6940.151443] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6940.151458] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6940.151462] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6940.151468] RAX: 00000000000039de RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6940.151471] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6940.151473] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6940.151476] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6940.151478] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6940.151482] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6940.151485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6940.151488] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6940.151497] Call Trace: [ 6940.151503] [ 6940.151508] ? show_regs.cold.9+0x22/0x2f [ 6940.154416] cpus=0-3 flags=0x4 nice=0 stuck for 4915s! [ 6940.154429] Showing busy workqueues and worker pools: [ 6940.154432] workqueue events: flags=0x0 [ 6940.154434] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6940.154443] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6940.154718] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6940.154727] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6940.155022] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6940.155033] pending: kfree_rcu_monitor [ 6940.155042] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6940.155049] pending: vmstat_shepherd, nfs4_renew_state [nfsv4] [ 6940.165619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6940.165640] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6940.165645] ? watchdog+0x40/0x40 [ 6940.165650] ? hrtimer_interrupt+0x128/0x2c0 [ 6940.165653] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6940.165662] ? apic_timer_interrupt+0xf/0x20 [ 6940.165667] [ 6940.165672] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6940.165684] do_raw_spin_lock+0x177/0x190 [ 6940.165753] _raw_spin_lock+0x21/0x30 [ 6940.165761] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6940.166305] ? do_raw_spin_unlock+0x75/0x190 [ 6940.166312] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6940.166681] ? do_wait_intr+0xf0/0xf0 [ 6940.166755] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6940.167023] mdc_close+0x53b/0xd70 [mdc] [ 6940.167129] lmv_close+0x18d/0x490 [lmv] [ 6940.167160] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6940.173003] , ldlm_pools_recalc_task [ptlrpc] [ 6940.175267] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6940.175548] ll_md_close+0x274/0x800 [lustre] [ 6940.175728] ll_file_release+0x128/0xc50 [lustre] [ 6940.175814] __fput+0x10d/0x440 [ 6940.175824] ____fput+0x12/0x20 [ 6940.175835] task_work_run+0xc6/0x110 [ 6940.175843] exit_to_usermode_loop+0x1dd/0x1f0 [ 6940.175850] do_syscall_64+0x3d6/0x440 [ 6940.175854] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6940.175861] RIP: 0033:0x7fdc0afbfcd8 [ 6940.175868] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6940.175872] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6940.175877] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6940.175879] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6940.175881] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6940.175883] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6940.175885] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6940.411284] , sec_gc_main [ptlrpc] [ 6940.412571] workqueue events_long: flags=0x0 [ 6940.416335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6940.432892] pending: lu_objects_flush [obdclass] [ 6940.434674] workqueue events_unbound: flags=0x2 [ 6940.442475] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6940.448931] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6940.458494] workqueue events_power_efficient: flags=0x80 [ 6940.460864] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6940.464645] pending: neigh_periodic_work, check_lifetime [ 6940.471824] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6940.482863] pending: do_cache_clean, neigh_periodic_work [ 6940.492862] workqueue mm_percpu_wq: flags=0x8 [ 6940.497224] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6940.499760] pending: lru_add_drain_per_cpu [ 6940.500920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6940.506883] pending: vmstat_update, lru_add_drain_per_cpu [ 6940.514166] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6940.517785] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6940.521504] workqueue writeback: flags=0x4a [ 6940.524967] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6940.527060] pending: wb_workfn, wb_workfn [ 6940.531545] workqueue xprtiod: flags=0xa [ 6940.534323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6940.539127] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6940.546388] workqueue ipv6_addrconf: flags=0x40008 [ 6940.548770] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6940.556182] pending: addrconf_verify_work [ 6940.559077] workqueue knbd0-recv: flags=0x1a [ 6940.562321] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6940.566380] in-flight: 655:recv_work [ 6940.568675] workqueue ptlrpc_pinger: flags=0x2 [ 6940.571015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6940.575145] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6940.578016] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4915s workers=4 idle: 36396 38 11188 [ 6940.587285] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6931s workers=2 idle: 658 [ 6952.052026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6952.055005] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6952.074165] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6952.078128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6952.080939] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6952.082478] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6952.088838] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6952.091668] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6952.094168] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6952.097168] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6952.101012] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6952.103545] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6952.106830] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6952.110289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6952.112428] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6952.116134] Call Trace: [ 6952.116952] [ 6952.117607] ? show_regs.cold.9+0x22/0x2f [ 6952.119180] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6952.120675] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6952.122427] ? watchdog+0x40/0x40 [ 6952.123573] ? hrtimer_interrupt+0x128/0x2c0 [ 6952.125491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6952.127206] ? apic_timer_interrupt+0xf/0x20 [ 6952.128541] [ 6952.129398] ? native_safe_halt+0x2e/0x50 [ 6952.130712] kvm_wait+0xcd/0xe0 [ 6952.132113] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6952.134342] do_raw_spin_lock+0x177/0x190 [ 6952.135617] _raw_spin_lock+0x21/0x30 [ 6952.136977] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6952.139536] ? do_raw_spin_unlock+0x75/0x190 [ 6952.141405] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6952.143447] ? do_wait_intr+0xf0/0xf0 [ 6952.144706] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6952.146830] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6952.148961] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6952.151323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6952.153317] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6952.155410] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6952.157026] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6952.158635] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6952.160688] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6952.163281] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6952.165936] lmv_intent_open+0x338/0x1150 [lmv] [ 6952.167614] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6952.169908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6952.172224] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6952.173719] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6952.176313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6952.178613] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6952.180524] ? do_raw_spin_unlock+0x75/0x190 [ 6952.181989] ? _raw_spin_unlock+0x12/0x30 [ 6952.183839] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6952.185709] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6952.188302] ll_atomic_open+0x36a/0xd70 [lustre] [ 6952.190489] lookup_open+0x356/0xb40 [ 6952.191718] do_last+0x313/0xfc0 [ 6952.193044] ? nd_jump_root+0xe5/0x160 [ 6952.194420] ? path_init+0x437/0x520 [ 6952.195693] path_openat+0xf7/0x500 [ 6952.197331] ? do_raw_spin_unlock+0x75/0x190 [ 6952.199282] do_filp_open+0x99/0x140 [ 6952.201161] ? getname_flags+0x6e/0x330 [ 6952.203197] ? __check_object_size+0xff/0x256 [ 6952.205202] ? do_raw_spin_unlock+0x75/0x190 [ 6952.207101] ? _raw_spin_unlock+0x12/0x30 [ 6952.208714] do_sys_openat2+0x2b4/0x410 [ 6952.209757] do_sys_open+0x73/0xa0 [ 6952.210475] __x64_sys_openat+0x24/0x30 [ 6952.211536] do_syscall_64+0xc1/0x440 [ 6952.213279] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6952.215178] RIP: 0033:0x7fdc0afbf332 [ 6952.216310] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6952.222293] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6952.224583] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6952.227796] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6952.229957] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6952.232770] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6952.236388] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6964.051013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6964.055818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6964.071140] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6964.074200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6964.077074] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6964.079542] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6964.081350] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6964.086500] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6964.088482] RAX: 0000000000007b74 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6964.090293] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6964.092195] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6964.094431] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6964.096408] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6964.098731] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6964.100983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6964.102675] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6964.104478] Call Trace: [ 6964.105180] [ 6964.105587] ? show_regs.cold.9+0x22/0x2f [ 6964.106691] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6964.108097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6964.109380] ? watchdog+0x40/0x40 [ 6964.110339] ? hrtimer_interrupt+0x128/0x2c0 [ 6964.111401] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6964.112553] ? apic_timer_interrupt+0xf/0x20 [ 6964.114001] [ 6964.114436] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6964.115988] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6964.117500] do_raw_spin_lock+0x177/0x190 [ 6964.119031] _raw_spin_lock+0x21/0x30 [ 6964.120202] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6964.121763] process_one_work+0x2c8/0x700 [ 6964.123043] worker_thread+0x296/0x6e0 [ 6964.124183] ? rescuer_thread+0x570/0x570 [ 6964.125185] kthread+0x1d1/0x200 [ 6964.126053] ? set_kthread_struct+0x70/0x70 [ 6964.127911] ret_from_fork+0x1f/0x30 [ 6968.048058] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6968.054373] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6968.064981] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6968.082522] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6968.095396] Modules linked in: [ 6968.100220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6968.100231] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6968.100245] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6968.100250] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6968.100255] RAX: 0000000000004592 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6968.100257] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6968.100259] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6968.100262] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6968.100264] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6968.100267] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6968.100270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6968.100273] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6968.100280] Call Trace: [ 6968.100286] [ 6968.100291] ? show_regs.cold.9+0x22/0x2f [ 6968.100929] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6968.100936] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6968.100940] ? watchdog+0x40/0x40 [ 6968.100945] ? hrtimer_interrupt+0x128/0x2c0 [ 6968.100950] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6968.100958] ? apic_timer_interrupt+0xf/0x20 [ 6968.100963] [ 6968.100966] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6968.100972] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6968.100976] ? dump_stack+0xf6/0x10e [ 6968.100982] do_raw_spin_lock+0x177/0x190 [ 6968.100987] _raw_spin_lock+0x21/0x30 [ 6968.100995] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6968.101276] obd_mod_free+0x46/0x140 [ptlrpc] [ 6968.101527] mdc_commit_open+0x91/0xb0 [mdc] [ 6968.101564] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6968.101765] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6968.101967] after_reply+0xe2c/0x1780 [ptlrpc] [ 6968.102168] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6968.107781] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6968.107964] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6968.107971] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6968.107974] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6968.107992] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6968.108001] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6968.108006] RAX: 0000000000000e0a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6968.108008] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6968.108010] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6968.108016] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6968.108019] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6968.108023] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6968.108026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6968.108028] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6968.108039] Call Trace: [ 6968.108045] [ 6968.108056] ? show_regs.cold.9+0x22/0x2f [ 6968.108072] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6968.108079] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6968.108089] ? watchdog+0x40/0x40 [ 6968.108094] ? hrtimer_interrupt+0x128/0x2c0 [ 6968.108103] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6968.108113] ? apic_timer_interrupt+0xf/0x20 [ 6968.108123] [ 6968.108127] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6968.108139] do_raw_spin_lock+0x177/0x190 [ 6968.108145] _raw_spin_lock+0x21/0x30 [ 6968.108154] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6968.112328] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6968.112714] ? do_wait_intr+0xf0/0xf0 [ 6968.112725] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6968.112979] kthread+0x1d1/0x200 [ 6968.112985] ? set_kthread_struct+0x70/0x70 [ 6968.112990] ret_from_fork+0x1f/0x30 [ 6968.430975] ? do_raw_spin_unlock+0x75/0x190 [ 6968.436392] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6968.440621] ? do_wait_intr+0xf0/0xf0 [ 6968.441928] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6968.444562] mdc_close+0x53b/0xd70 [mdc] [ 6968.450974] lmv_close+0x18d/0x490 [lmv] [ 6968.456075] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6968.469326] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6968.471238] ll_md_close+0x274/0x800 [lustre] [ 6968.476081] ll_file_release+0x128/0xc50 [lustre] [ 6968.478358] __fput+0x10d/0x440 [ 6968.486001] ____fput+0x12/0x20 [ 6968.491225] task_work_run+0xc6/0x110 [ 6968.492650] exit_to_usermode_loop+0x1dd/0x1f0 [ 6968.494185] do_syscall_64+0x3d6/0x440 [ 6968.501780] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6968.505240] RIP: 0033:0x7fdc0afbfcd8 [ 6968.507368] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6968.532357] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6968.540448] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6968.545295] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6968.553907] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6968.561964] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6968.566755] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6970.847063] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4945s! [ 6970.855590] Showing busy workqueues and worker pools: [ 6970.858583] workqueue events: flags=0x0 [ 6970.860743] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6970.864761] pending: cpu_latency_work [ptlrpc], push_to_pool [ 6970.868360] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6970.871613] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 6970.876010] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6970.878962] pending: kfree_rcu_monitor [ 6970.880841] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6970.884265] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 6970.898253] workqueue events_long: flags=0x0 [ 6970.904521] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6970.912176] pending: lu_objects_flush [obdclass] [ 6970.915530] workqueue events_unbound: flags=0x2 [ 6970.918031] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6970.921470] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6970.928946] workqueue events_power_efficient: flags=0x80 [ 6970.932093] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6970.936051] pending: neigh_periodic_work, check_lifetime [ 6970.939097] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6970.942841] pending: do_cache_clean, neigh_periodic_work [ 6970.945963] workqueue mm_percpu_wq: flags=0x8 [ 6970.948356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6970.951898] pending: lru_add_drain_per_cpu [ 6970.956510] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6970.963536] pending: vmstat_update, lru_add_drain_per_cpu [ 6970.966615] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6970.970463] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6970.978027] workqueue writeback: flags=0x4a [ 6970.981911] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6970.988349] pending: wb_workfn, wb_workfn [ 6970.993514] workqueue xprtiod: flags=0xa [ 6970.996146] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6971.002907] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6971.011896] workqueue ipv6_addrconf: flags=0x40008 [ 6971.018994] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6971.025376] pending: addrconf_verify_work [ 6971.029359] workqueue knbd0-recv: flags=0x1a [ 6971.034041] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6971.037259] in-flight: 655:recv_work [ 6971.039280] workqueue ptlrpc_pinger: flags=0x2 [ 6971.042036] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6971.044906] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 6971.047975] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4946s workers=4 idle: 36396 38 11188 [ 6971.052013] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6961s workers=2 idle: 658 [ 6980.052056] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 6980.056406] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6980.079981] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6980.085479] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6980.089211] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6980.091182] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6980.101191] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6980.107876] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 6980.113640] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 6980.116906] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 6980.120838] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 6980.123732] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6980.129229] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 6980.141137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6980.146589] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 6980.151668] Call Trace: [ 6980.153364] [ 6980.154572] ? show_regs.cold.9+0x22/0x2f [ 6980.156362] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6980.159764] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6980.162771] ? watchdog+0x40/0x40 [ 6980.164871] ? hrtimer_interrupt+0x128/0x2c0 [ 6980.167193] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6980.170702] ? apic_timer_interrupt+0xf/0x20 [ 6980.173435] [ 6980.174577] ? native_safe_halt+0x2e/0x50 [ 6980.176974] kvm_wait+0xcd/0xe0 [ 6980.179407] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6980.182430] do_raw_spin_lock+0x177/0x190 [ 6980.184384] _raw_spin_lock+0x21/0x30 [ 6980.186338] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6980.189968] ? do_raw_spin_unlock+0x75/0x190 [ 6980.192777] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6980.195797] ? do_wait_intr+0xf0/0xf0 [ 6980.197786] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6980.202212] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6980.206743] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6980.212217] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6980.216128] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6980.220424] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6980.222454] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6980.224441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6980.226933] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6980.232394] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6980.236990] lmv_intent_open+0x338/0x1150 [lmv] [ 6980.241737] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6980.244782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6980.247698] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6980.249842] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6980.252721] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6980.255092] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6980.257826] ? do_raw_spin_unlock+0x75/0x190 [ 6980.259829] ? _raw_spin_unlock+0x12/0x30 [ 6980.262461] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6980.264884] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6980.267485] ll_atomic_open+0x36a/0xd70 [lustre] [ 6980.269991] lookup_open+0x356/0xb40 [ 6980.271825] do_last+0x313/0xfc0 [ 6980.273455] ? nd_jump_root+0xe5/0x160 [ 6980.274891] ? path_init+0x437/0x520 [ 6980.276764] path_openat+0xf7/0x500 [ 6980.278392] ? do_raw_spin_unlock+0x75/0x190 [ 6980.280204] do_filp_open+0x99/0x140 [ 6980.281900] ? getname_flags+0x6e/0x330 [ 6980.283889] ? __check_object_size+0xff/0x256 [ 6980.286454] ? do_raw_spin_unlock+0x75/0x190 [ 6980.288448] ? _raw_spin_unlock+0x12/0x30 [ 6980.290494] do_sys_openat2+0x2b4/0x410 [ 6980.292108] do_sys_open+0x73/0xa0 [ 6980.293237] __x64_sys_openat+0x24/0x30 [ 6980.295423] do_syscall_64+0xc1/0x440 [ 6980.297637] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6980.299974] RIP: 0033:0x7fdc0afbf332 [ 6980.302086] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6980.312545] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6980.316566] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 6980.320045] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 6980.323670] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 6980.327348] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 6980.334239] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 6992.051275] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 6992.058022] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6992.095642] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6992.101827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6992.108623] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6992.113014] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6992.115962] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6992.128255] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6992.132467] RAX: 000000000000740b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6992.135324] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 6992.137287] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6992.140224] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 6992.144298] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 6992.146093] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 6992.155720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6992.164778] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 6992.169557] Call Trace: [ 6992.170350] [ 6992.171926] ? show_regs.cold.9+0x22/0x2f [ 6992.173385] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6992.175237] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6992.178668] ? watchdog+0x40/0x40 [ 6992.181491] ? hrtimer_interrupt+0x128/0x2c0 [ 6992.186366] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6992.188031] ? apic_timer_interrupt+0xf/0x20 [ 6992.191332] [ 6992.192285] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6992.195021] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6992.200325] do_raw_spin_lock+0x177/0x190 [ 6992.205470] _raw_spin_lock+0x21/0x30 [ 6992.207882] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6992.210548] process_one_work+0x2c8/0x700 [ 6992.212187] worker_thread+0x296/0x6e0 [ 6992.214916] ? rescuer_thread+0x570/0x570 [ 6992.218602] kthread+0x1d1/0x200 [ 6992.220533] ? set_kthread_struct+0x70/0x70 [ 6992.223478] ret_from_fork+0x1f/0x30 [ 6996.048424] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 6996.054027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 6996.057074] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 6996.060479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6996.060562] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6996.060569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6996.060572] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6996.060585] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6996.060589] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6996.060668] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6996.060671] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 6996.060673] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6996.060675] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 6996.060678] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 6996.060681] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 6996.060685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6996.060687] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 6996.060702] Call Trace: [ 6996.060707] [ 6996.060712] ? show_regs.cold.9+0x22/0x2f [ 6996.060722] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6996.060728] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6996.060733] ? watchdog+0x40/0x40 [ 6996.060737] ? hrtimer_interrupt+0x128/0x2c0 [ 6996.060742] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6996.060748] ? apic_timer_interrupt+0xf/0x20 [ 6996.060752] [ 6996.060756] ? native_safe_halt+0x2e/0x50 [ 6996.060760] kvm_wait+0xcd/0xe0 [ 6996.060766] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6996.060774] do_raw_spin_lock+0x177/0x190 [ 6996.060778] _raw_spin_lock+0x21/0x30 [ 6996.060783] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6996.061187] ? do_raw_spin_unlock+0x75/0x190 [ 6996.061197] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6996.061467] ? do_wait_intr+0xf0/0xf0 [ 6996.061474] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6996.061750] mdc_close+0x53b/0xd70 [mdc] [ 6996.061803] lmv_close+0x18d/0x490 [lmv] [ 6996.068259] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror [ 6996.097368] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6996.109386] dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6996.109414] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6996.109421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6996.109425] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6996.109437] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6996.109441] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6996.109446] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 6996.109448] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 6996.109451] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 6996.109453] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 6996.109455] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 6996.109459] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 6996.109461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6996.109463] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 6996.109471] Call Trace: [ 6996.109476] [ 6996.109482] ? show_regs.cold.9+0x22/0x2f [ 6996.109491] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6996.109498] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6996.109502] ? watchdog+0x40/0x40 [ 6996.109507] ? hrtimer_interrupt+0x128/0x2c0 [ 6996.109511] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6996.109517] ? apic_timer_interrupt+0xf/0x20 [ 6996.109523] [ 6996.116031] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6996.119781] ? native_safe_halt+0x2e/0x50 [ 6996.119799] kvm_wait+0xcd/0xe0 [ 6996.119808] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6996.119816] ? dump_stack+0xf6/0x10e [ 6996.119822] do_raw_spin_lock+0x177/0x190 [ 6996.119827] _raw_spin_lock+0x21/0x30 [ 6996.119834] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6996.134578] ll_md_close+0x274/0x800 [lustre] [ 6996.134901] ll_file_release+0x128/0xc50 [lustre] [ 6996.135053] __fput+0x10d/0x440 [ 6996.135071] ____fput+0x12/0x20 [ 6996.135076] task_work_run+0xc6/0x110 [ 6996.135149] exit_to_usermode_loop+0x1dd/0x1f0 [ 6996.135157] do_syscall_64+0x3d6/0x440 [ 6996.135162] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6996.135176] RIP: 0033:0x7fdc0afbfcd8 [ 6996.135183] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6996.135187] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6996.135313] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 6996.135318] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 6996.135320] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 6996.135323] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 6996.135325] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6996.418556] obd_mod_free+0x46/0x140 [ptlrpc] [ 6996.421826] mdc_commit_open+0x91/0xb0 [mdc] [ 6996.423641] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6996.430741] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6996.439607] after_reply+0xe2c/0x1780 [ptlrpc] [ 6996.441885] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6996.444883] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6996.448108] ? do_wait_intr+0xf0/0xf0 [ 6996.454512] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6996.458567] kthread+0x1d1/0x200 [ 6996.461619] ? set_kthread_struct+0x70/0x70 [ 6996.463190] ret_from_fork+0x1f/0x30 [ 7001.569388] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4976s! [ 7001.579953] Showing busy workqueues and worker pools: [ 7001.585268] workqueue events: flags=0x0 [ 7001.587977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7001.592796] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7001.600138] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7001.604893] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7001.617223] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7001.620210] pending: kfree_rcu_monitor [ 7001.622627] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7001.627828] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7001.635902] workqueue events_long: flags=0x0 [ 7001.636984] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7001.640639] pending: lu_objects_flush [obdclass] [ 7001.644076] workqueue events_unbound: flags=0x2 [ 7001.646167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7001.649053] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7001.655234] workqueue events_power_efficient: flags=0x80 [ 7001.656580] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7001.660073] pending: neigh_periodic_work, check_lifetime [ 7001.661619] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7001.663719] pending: do_cache_clean, neigh_periodic_work [ 7001.665306] workqueue mm_percpu_wq: flags=0x8 [ 7001.666558] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7001.668419] pending: lru_add_drain_per_cpu [ 7001.669657] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7001.671551] pending: vmstat_update, lru_add_drain_per_cpu [ 7001.673662] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7001.676285] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7001.678696] workqueue writeback: flags=0x4a [ 7001.680688] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7001.683532] pending: wb_workfn, wb_workfn [ 7001.685572] workqueue xprtiod: flags=0xa [ 7001.687267] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7001.689782] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7001.692824] workqueue ipv6_addrconf: flags=0x40008 [ 7001.700571] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7001.703648] pending: addrconf_verify_work [ 7001.708597] workqueue knbd0-recv: flags=0x1a [ 7001.712464] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7001.719325] in-flight: 655:recv_work [ 7001.721302] workqueue ptlrpc_pinger: flags=0x2 [ 7001.723658] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7001.729673] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7001.732065] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4976s workers=4 idle: 36396 38 11188 [ 7001.746367] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6992s workers=2 idle: 658 [ 7008.059831] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7008.075615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7008.130214] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7008.154094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7008.177172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7008.192530] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7008.221130] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7008.224885] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7008.235941] RDX: 000000000000670c RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7008.244074] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7008.250148] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7008.260659] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7008.271035] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7008.281262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7008.285426] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7008.296608] Call Trace: [ 7008.298668] [ 7008.302261] ? show_regs.cold.9+0x22/0x2f [ 7008.311058] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7008.314744] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7008.316140] ? watchdog+0x40/0x40 [ 7008.325596] ? hrtimer_interrupt+0x128/0x2c0 [ 7008.330884] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7008.343476] ? apic_timer_interrupt+0xf/0x20 [ 7008.347247] [ 7008.351677] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7008.358366] do_raw_spin_lock+0x177/0x190 [ 7008.361452] _raw_spin_lock+0x21/0x30 [ 7008.365974] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7008.370537] ? do_raw_spin_unlock+0x75/0x190 [ 7008.373330] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7008.381705] ? do_wait_intr+0xf0/0xf0 [ 7008.385039] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7008.391995] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7008.395464] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7008.399267] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7008.402248] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7008.410426] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7008.416334] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7008.419087] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7008.422227] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7008.426312] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7008.430325] lmv_intent_open+0x338/0x1150 [lmv] [ 7008.433326] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7008.443022] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7008.446690] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7008.451914] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7008.455116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7008.465608] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7008.468525] ? do_raw_spin_unlock+0x75/0x190 [ 7008.470039] ? _raw_spin_unlock+0x12/0x30 [ 7008.471970] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7008.474054] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7008.476358] ll_atomic_open+0x36a/0xd70 [lustre] [ 7008.478908] lookup_open+0x356/0xb40 [ 7008.480300] do_last+0x313/0xfc0 [ 7008.481534] ? nd_jump_root+0xe5/0x160 [ 7008.482936] ? path_init+0x437/0x520 [ 7008.484380] path_openat+0xf7/0x500 [ 7008.487269] ? do_raw_spin_unlock+0x75/0x190 [ 7008.488902] do_filp_open+0x99/0x140 [ 7008.489977] ? getname_flags+0x6e/0x330 [ 7008.494000] ? __check_object_size+0xff/0x256 [ 7008.495946] ? do_raw_spin_unlock+0x75/0x190 [ 7008.499723] ? _raw_spin_unlock+0x12/0x30 [ 7008.501566] do_sys_openat2+0x2b4/0x410 [ 7008.504408] do_sys_open+0x73/0xa0 [ 7008.508226] __x64_sys_openat+0x24/0x30 [ 7008.509518] do_syscall_64+0xc1/0x440 [ 7008.511547] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7008.513524] RIP: 0033:0x7fdc0afbf332 [ 7008.516979] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7008.526685] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7008.530940] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7008.533965] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7008.538443] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7008.541775] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7008.547639] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7020.051136] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7020.059095] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7020.096854] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7020.105226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7020.114039] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7020.118222] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7020.123985] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7020.147258] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7020.151455] RAX: 000000000000585c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7020.154932] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7020.162062] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7020.166471] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7020.168064] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7020.172022] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7020.179531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7020.181703] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7020.186158] Call Trace: [ 7020.189449] [ 7020.190262] ? show_regs.cold.9+0x22/0x2f [ 7020.192664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7020.194268] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7020.196375] ? watchdog+0x40/0x40 [ 7020.197428] ? hrtimer_interrupt+0x128/0x2c0 [ 7020.198809] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7020.205982] ? apic_timer_interrupt+0xf/0x20 [ 7020.207380] [ 7020.213733] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7020.219034] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7020.224272] do_raw_spin_lock+0x177/0x190 [ 7020.229046] _raw_spin_lock+0x21/0x30 [ 7020.230273] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7020.232694] process_one_work+0x2c8/0x700 [ 7020.233838] worker_thread+0x296/0x6e0 [ 7020.238320] ? rescuer_thread+0x570/0x570 [ 7020.240332] kthread+0x1d1/0x200 [ 7020.241907] ? set_kthread_struct+0x70/0x70 [ 7020.244464] ret_from_fork+0x1f/0x30 [ 7024.054021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7024.057187] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7024.076394] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7024.084158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7024.088043] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7024.089848] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7024.102019] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7024.107099] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7024.110091] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7024.113763] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7024.117019] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7024.119642] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7024.123033] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7024.127427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7024.133279] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7024.138696] Call Trace: [ 7024.139657] [ 7024.140407] ? show_regs.cold.9+0x22/0x2f [ 7024.142179] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7024.144602] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7024.146690] ? watchdog+0x40/0x40 [ 7024.148449] ? hrtimer_interrupt+0x128/0x2c0 [ 7024.150358] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7024.155826] ? apic_timer_interrupt+0xf/0x20 [ 7024.159173] [ 7024.163693] ? native_safe_halt+0x2e/0x50 [ 7024.165959] kvm_wait+0xcd/0xe0 [ 7024.169734] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7024.172388] do_raw_spin_lock+0x177/0x190 [ 7024.174728] _raw_spin_lock+0x21/0x30 [ 7024.179561] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7024.187239] ? do_raw_spin_unlock+0x75/0x190 [ 7024.196682] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7024.198673] ? do_wait_intr+0xf0/0xf0 [ 7024.203388] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7024.210161] mdc_close+0x53b/0xd70 [mdc] [ 7024.211690] lmv_close+0x18d/0x490 [lmv] [ 7024.213725] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7024.217861] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7024.220790] ll_md_close+0x274/0x800 [lustre] [ 7024.225130] ll_file_release+0x128/0xc50 [lustre] [ 7024.229866] __fput+0x10d/0x440 [ 7024.236134] ____fput+0x12/0x20 [ 7024.237301] task_work_run+0xc6/0x110 [ 7024.238670] exit_to_usermode_loop+0x1dd/0x1f0 [ 7024.244812] do_syscall_64+0x3d6/0x440 [ 7024.246198] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7024.257413] RIP: 0033:0x7fdc0afbfcd8 [ 7024.262782] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7024.286658] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7024.296117] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7024.302864] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7024.324024] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7024.342097] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7024.350517] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7028.048130] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7028.051002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7028.077132] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7028.083136] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7028.088309] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7028.090859] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7028.099791] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7028.103289] RAX: 0000000000005020 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7028.106823] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7028.110135] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7028.113129] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7028.117123] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7028.120473] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7028.124665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7028.128124] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7028.131546] Call Trace: [ 7028.133041] [ 7028.134222] ? show_regs.cold.9+0x22/0x2f [ 7028.135806] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7028.138392] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7028.140552] ? watchdog+0x40/0x40 [ 7028.142355] ? hrtimer_interrupt+0x128/0x2c0 [ 7028.144254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7028.146916] ? apic_timer_interrupt+0xf/0x20 [ 7028.149303] [ 7028.151074] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7028.153887] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7028.156821] ? dump_stack+0xf6/0x10e [ 7028.158750] do_raw_spin_lock+0x177/0x190 [ 7028.160829] _raw_spin_lock+0x21/0x30 [ 7028.162650] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7028.165318] obd_mod_free+0x46/0x140 [ptlrpc] [ 7028.168365] mdc_commit_open+0x91/0xb0 [mdc] [ 7028.170720] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7028.174470] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7028.178124] after_reply+0xe2c/0x1780 [ptlrpc] [ 7028.180990] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7028.183923] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7028.185519] ? do_wait_intr+0xf0/0xf0 [ 7028.186477] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7028.189043] kthread+0x1d1/0x200 [ 7028.190703] ? set_kthread_struct+0x70/0x70 [ 7028.192693] ret_from_fork+0x1f/0x30 [ 7031.775036] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5006s! [ 7031.777227] Showing busy workqueues and worker pools: [ 7031.780610] workqueue events: flags=0x0 [ 7031.782203] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7031.784153] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7031.787983] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7031.790491] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7031.794334] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7031.796741] pending: kfree_rcu_monitor [ 7031.798034] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7031.800453] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7031.804672] workqueue events_long: flags=0x0 [ 7031.805935] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7031.811728] pending: lu_objects_flush [obdclass] [ 7031.813485] workqueue events_unbound: flags=0x2 [ 7031.815899] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7031.818932] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7031.822933] workqueue events_power_efficient: flags=0x80 [ 7031.827152] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7031.835948] pending: neigh_periodic_work, check_lifetime [ 7031.839506] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7031.846113] pending: do_cache_clean, neigh_periodic_work [ 7031.848526] workqueue mm_percpu_wq: flags=0x8 [ 7031.849824] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7031.852410] pending: lru_add_drain_per_cpu [ 7031.853819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7031.859961] pending: vmstat_update, lru_add_drain_per_cpu [ 7031.865698] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7031.868607] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7031.871417] workqueue writeback: flags=0x4a [ 7031.873471] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7031.875844] pending: wb_workfn, wb_workfn [ 7031.877335] workqueue xprtiod: flags=0xa [ 7031.880074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7031.881636] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7031.889708] workqueue ipv6_addrconf: flags=0x40008 [ 7031.891646] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7031.894492] pending: addrconf_verify_work [ 7031.896602] workqueue knbd0-recv: flags=0x1a [ 7031.898431] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7031.900940] in-flight: 655:recv_work [ 7031.902518] workqueue ptlrpc_pinger: flags=0x2 [ 7031.904483] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7031.906610] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7031.909067] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5006s workers=4 idle: 36396 38 11188 [ 7031.912515] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7022s workers=2 idle: 658 [ 7036.052842] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7036.055632] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7036.074571] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7036.079128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7036.082987] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7036.085011] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7036.091971] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7036.094501] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7036.097022] RDX: 0000000000001b81 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7036.099477] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7036.102342] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7036.104907] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7036.107749] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7036.110555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7036.112452] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7036.114986] Call Trace: [ 7036.115754] [ 7036.116605] ? show_regs.cold.9+0x22/0x2f [ 7036.118018] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7036.120021] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7036.121401] ? watchdog+0x40/0x40 [ 7036.122822] ? hrtimer_interrupt+0x128/0x2c0 [ 7036.124213] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7036.126138] ? apic_timer_interrupt+0xf/0x20 [ 7036.127533] [ 7036.128331] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7036.130343] do_raw_spin_lock+0x177/0x190 [ 7036.131604] _raw_spin_lock+0x21/0x30 [ 7036.133364] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7036.135170] ? do_raw_spin_unlock+0x75/0x190 [ 7036.136949] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7036.139070] ? do_wait_intr+0xf0/0xf0 [ 7036.140251] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7036.142572] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7036.144505] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7036.146987] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7036.149057] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7036.151388] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7036.152764] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7036.154339] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7036.156253] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7036.158821] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7036.161298] lmv_intent_open+0x338/0x1150 [lmv] [ 7036.162962] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7036.165114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7036.166984] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7036.168331] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7036.170570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7036.172590] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7036.174197] ? do_raw_spin_unlock+0x75/0x190 [ 7036.175930] ? _raw_spin_unlock+0x12/0x30 [ 7036.177251] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7036.179304] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7036.181087] ll_atomic_open+0x36a/0xd70 [lustre] [ 7036.183104] lookup_open+0x356/0xb40 [ 7036.184607] do_last+0x313/0xfc0 [ 7036.186070] ? nd_jump_root+0xe5/0x160 [ 7036.187750] ? path_init+0x437/0x520 [ 7036.189166] path_openat+0xf7/0x500 [ 7036.190646] ? do_raw_spin_unlock+0x75/0x190 [ 7036.192075] do_filp_open+0x99/0x140 [ 7036.193828] ? getname_flags+0x6e/0x330 [ 7036.195138] ? __check_object_size+0xff/0x256 [ 7036.197099] ? do_raw_spin_unlock+0x75/0x190 [ 7036.198703] ? _raw_spin_unlock+0x12/0x30 [ 7036.200069] do_sys_openat2+0x2b4/0x410 [ 7036.201701] do_sys_open+0x73/0xa0 [ 7036.202943] __x64_sys_openat+0x24/0x30 [ 7036.204616] do_syscall_64+0xc1/0x440 [ 7036.206235] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7036.208036] RIP: 0033:0x7fdc0afbf332 [ 7036.209516] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7036.216305] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7036.218761] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7036.220954] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7036.224801] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7036.227842] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7036.230293] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7048.051030] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7048.055248] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7048.095436] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7048.100947] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7048.105530] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7048.110647] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7048.112473] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7048.126665] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7048.138228] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7048.142862] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7048.147936] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7048.152862] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7048.158908] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7048.166335] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7048.172899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7048.176954] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7048.182192] Call Trace: [ 7048.183332] [ 7048.184769] ? show_regs.cold.9+0x22/0x2f [ 7048.192938] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7048.196814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7048.202980] ? watchdog+0x40/0x40 [ 7048.204285] ? hrtimer_interrupt+0x128/0x2c0 [ 7048.205814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7048.218057] ? apic_timer_interrupt+0xf/0x20 [ 7048.219605] [ 7048.220307] ? native_safe_halt+0x2e/0x50 [ 7048.234996] kvm_wait+0xcd/0xe0 [ 7048.240716] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7048.246738] do_raw_spin_lock+0x177/0x190 [ 7048.248901] _raw_spin_lock+0x21/0x30 [ 7048.250261] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7048.258784] process_one_work+0x2c8/0x700 [ 7048.265907] worker_thread+0x296/0x6e0 [ 7048.277843] ? rescuer_thread+0x570/0x570 [ 7048.280333] kthread+0x1d1/0x200 [ 7048.284365] ? set_kthread_struct+0x70/0x70 [ 7048.296963] ret_from_fork+0x1f/0x30 [ 7052.054095] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7052.067315] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7052.117230] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7052.132654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7052.145208] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7052.150324] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7052.171752] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7052.180632] RAX: 0000000000004411 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7052.191171] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7052.195364] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7052.203809] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7052.213975] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7052.224417] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7052.233739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7052.238133] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7052.245255] Call Trace: [ 7052.255311] [ 7052.261728] ? show_regs.cold.9+0x22/0x2f [ 7052.273105] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7052.277268] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7052.281514] ? watchdog+0x40/0x40 [ 7052.284824] ? hrtimer_interrupt+0x128/0x2c0 [ 7052.286893] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7052.298091] ? apic_timer_interrupt+0xf/0x20 [ 7052.307152] [ 7052.308143] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7052.313825] do_raw_spin_lock+0x177/0x190 [ 7052.317065] _raw_spin_lock+0x21/0x30 [ 7052.324816] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7052.329672] ? do_raw_spin_unlock+0x75/0x190 [ 7052.332308] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7052.338013] ? do_wait_intr+0xf0/0xf0 [ 7052.340553] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7052.350089] mdc_close+0x53b/0xd70 [mdc] [ 7052.352779] lmv_close+0x18d/0x490 [lmv] [ 7052.354227] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7052.367049] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7052.371673] ll_md_close+0x274/0x800 [lustre] [ 7052.374312] ll_file_release+0x128/0xc50 [lustre] [ 7052.386077] __fput+0x10d/0x440 [ 7052.390592] ____fput+0x12/0x20 [ 7052.396215] task_work_run+0xc6/0x110 [ 7052.400183] exit_to_usermode_loop+0x1dd/0x1f0 [ 7052.402097] do_syscall_64+0x3d6/0x440 [ 7052.406164] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7052.409684] RIP: 0033:0x7fdc0afbfcd8 [ 7052.426427] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7052.448828] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7052.462798] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7052.469269] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7052.481406] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7052.492914] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7052.500887] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7056.050390] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7056.056318] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7056.080885] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7056.087248] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7056.103157] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7056.116050] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7056.135582] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7056.150452] RAX: 0000000000003691 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7056.161591] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7056.168890] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7056.173094] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7056.179393] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7056.185734] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7056.194098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7056.200954] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7056.206864] Call Trace: [ 7056.208177] [ 7056.211902] ? show_regs.cold.9+0x22/0x2f [ 7056.215420] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7056.220294] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7056.231098] ? watchdog+0x40/0x40 [ 7056.233482] ? hrtimer_interrupt+0x128/0x2c0 [ 7056.239786] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7056.242422] ? apic_timer_interrupt+0xf/0x20 [ 7056.246728] [ 7056.249469] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7056.254167] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7056.258438] ? dump_stack+0xf6/0x10e [ 7056.263397] do_raw_spin_lock+0x177/0x190 [ 7056.268845] _raw_spin_lock+0x21/0x30 [ 7056.271905] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7056.274654] obd_mod_free+0x46/0x140 [ptlrpc] [ 7056.277280] mdc_commit_open+0x91/0xb0 [mdc] [ 7056.279233] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7056.282343] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7056.290122] after_reply+0xe2c/0x1780 [ptlrpc] [ 7056.293148] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7056.300262] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7056.306568] ? do_wait_intr+0xf0/0xf0 [ 7056.311789] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7056.319002] kthread+0x1d1/0x200 [ 7056.321650] ? set_kthread_struct+0x70/0x70 [ 7056.323530] ret_from_fork+0x1f/0x30 [ 7061.992639] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5037s! [ 7062.003907] Showing busy workqueues and worker pools: [ 7062.014516] workqueue events: flags=0x0 [ 7062.020930] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7062.030012] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7062.035651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7062.039869] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7062.048093] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7062.051774] pending: kfree_rcu_monitor [ 7062.052837] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7062.060671] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7062.070489] workqueue events_long: flags=0x0 [ 7062.074534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7062.078433] pending: lu_objects_flush [obdclass] [ 7062.086783] workqueue events_unbound: flags=0x2 [ 7062.091151] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7062.098794] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7062.113460] workqueue events_power_efficient: flags=0x80 [ 7062.119135] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7062.126497] pending: neigh_periodic_work, check_lifetime [ 7062.131473] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7062.138418] pending: do_cache_clean, neigh_periodic_work [ 7062.141187] workqueue mm_percpu_wq: flags=0x8 [ 7062.143312] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7062.147043] pending: lru_add_drain_per_cpu [ 7062.151671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7062.158972] pending: vmstat_update, lru_add_drain_per_cpu [ 7062.164766] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7062.169678] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7062.176900] workqueue writeback: flags=0x4a [ 7062.178886] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7062.182175] pending: wb_workfn, wb_workfn [ 7062.189092] workqueue xprtiod: flags=0xa [ 7062.195150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7062.197632] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7062.200767] workqueue ipv6_addrconf: flags=0x40008 [ 7062.203123] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7062.206446] pending: addrconf_verify_work [ 7062.208782] workqueue knbd0-recv: flags=0x1a [ 7062.210913] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7062.213906] in-flight: 655:recv_work [ 7062.215736] workqueue ptlrpc_pinger: flags=0x2 [ 7062.217751] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7062.220379] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7062.222950] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5037s workers=4 idle: 36396 38 11188 [ 7062.226656] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7052s workers=2 idle: 658 [ 7064.052089] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7064.056586] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7064.092380] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7064.099727] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7064.103562] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7064.105802] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7064.113732] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7064.117281] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7064.120046] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7064.123096] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7064.125751] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7064.128260] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7064.131042] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7064.133657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7064.135571] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7064.139053] Call Trace: [ 7064.140281] [ 7064.141033] ? show_regs.cold.9+0x22/0x2f [ 7064.143441] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7064.145264] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7064.147135] ? watchdog+0x40/0x40 [ 7064.148767] ? hrtimer_interrupt+0x128/0x2c0 [ 7064.150483] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7064.152483] ? apic_timer_interrupt+0xf/0x20 [ 7064.154196] [ 7064.155026] ? native_safe_halt+0x2e/0x50 [ 7064.156735] kvm_wait+0xcd/0xe0 [ 7064.157898] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7064.160996] do_raw_spin_lock+0x177/0x190 [ 7064.162142] _raw_spin_lock+0x21/0x30 [ 7064.167712] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7064.170809] ? do_raw_spin_unlock+0x75/0x190 [ 7064.172896] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7064.198607] ? do_wait_intr+0xf0/0xf0 [ 7064.199962] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7064.207278] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7064.212731] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7064.216447] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7064.219095] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7064.224719] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7064.230724] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7064.235310] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7064.240882] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7064.247145] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7064.251650] lmv_intent_open+0x338/0x1150 [lmv] [ 7064.256243] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7064.260369] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7064.262891] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7064.264982] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7064.271632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7064.273535] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7064.284283] ? do_raw_spin_unlock+0x75/0x190 [ 7064.287261] ? _raw_spin_unlock+0x12/0x30 [ 7064.290191] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7064.292927] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7064.294933] ll_atomic_open+0x36a/0xd70 [lustre] [ 7064.297388] lookup_open+0x356/0xb40 [ 7064.301430] do_last+0x313/0xfc0 [ 7064.305928] ? nd_jump_root+0xe5/0x160 [ 7064.309171] ? path_init+0x437/0x520 [ 7064.318080] path_openat+0xf7/0x500 [ 7064.319361] ? do_raw_spin_unlock+0x75/0x190 [ 7064.323682] do_filp_open+0x99/0x140 [ 7064.324993] ? getname_flags+0x6e/0x330 [ 7064.328421] ? __check_object_size+0xff/0x256 [ 7064.330749] ? do_raw_spin_unlock+0x75/0x190 [ 7064.333278] ? _raw_spin_unlock+0x12/0x30 [ 7064.337078] do_sys_openat2+0x2b4/0x410 [ 7064.341359] do_sys_open+0x73/0xa0 [ 7064.344441] __x64_sys_openat+0x24/0x30 [ 7064.347660] do_syscall_64+0xc1/0x440 [ 7064.352603] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7064.358318] RIP: 0033:0x7fdc0afbf332 [ 7064.360887] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7064.371816] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7064.380627] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7064.387901] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7064.390335] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7064.396693] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7064.403323] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7076.054372] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7076.060445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7076.089646] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7076.099669] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7076.113678] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7076.116083] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7076.118175] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7076.125309] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7076.128195] RAX: 0000000000007d24 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7076.130747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7076.133418] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7076.136063] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7076.138255] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7076.140663] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7076.143319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7076.145525] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7076.148210] Call Trace: [ 7076.149099] [ 7076.150005] ? show_regs.cold.9+0x22/0x2f [ 7076.151496] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7076.153330] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7076.154867] ? watchdog+0x40/0x40 [ 7076.155912] ? hrtimer_interrupt+0x128/0x2c0 [ 7076.157442] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7076.158956] ? apic_timer_interrupt+0xf/0x20 [ 7076.160391] [ 7076.161105] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7076.162808] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7076.165195] do_raw_spin_lock+0x177/0x190 [ 7076.166737] _raw_spin_lock+0x21/0x30 [ 7076.168086] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7076.170461] process_one_work+0x2c8/0x700 [ 7076.172052] worker_thread+0x296/0x6e0 [ 7076.173804] ? rescuer_thread+0x570/0x570 [ 7076.175319] kthread+0x1d1/0x200 [ 7076.176760] ? set_kthread_struct+0x70/0x70 [ 7076.178156] ret_from_fork+0x1f/0x30 [ 7080.054031] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7080.057333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7080.095099] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7080.101040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7080.105747] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7080.108170] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7080.118310] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7080.122986] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7080.127321] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7080.131086] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7080.134973] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7080.139171] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7080.142588] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7080.146600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7080.149343] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7080.153758] Call Trace: [ 7080.156536] [ 7080.159478] ? show_regs.cold.9+0x22/0x2f [ 7080.161781] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7080.164543] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7080.168246] ? watchdog+0x40/0x40 [ 7080.170048] ? hrtimer_interrupt+0x128/0x2c0 [ 7080.172433] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7080.175790] ? apic_timer_interrupt+0xf/0x20 [ 7080.178775] [ 7080.180084] ? native_safe_halt+0x2e/0x50 [ 7080.182889] kvm_wait+0xcd/0xe0 [ 7080.185430] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7080.188330] do_raw_spin_lock+0x177/0x190 [ 7080.190838] _raw_spin_lock+0x21/0x30 [ 7080.192979] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7080.195989] ? do_raw_spin_unlock+0x75/0x190 [ 7080.198378] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7080.201522] ? do_wait_intr+0xf0/0xf0 [ 7080.204013] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7080.207309] mdc_close+0x53b/0xd70 [mdc] [ 7080.209522] lmv_close+0x18d/0x490 [lmv] [ 7080.211853] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7080.215498] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7080.218453] ll_md_close+0x274/0x800 [lustre] [ 7080.220698] ll_file_release+0x128/0xc50 [lustre] [ 7080.223465] __fput+0x10d/0x440 [ 7080.224896] ____fput+0x12/0x20 [ 7080.226542] task_work_run+0xc6/0x110 [ 7080.228476] exit_to_usermode_loop+0x1dd/0x1f0 [ 7080.230760] do_syscall_64+0x3d6/0x440 [ 7080.232873] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7080.235543] RIP: 0033:0x7fdc0afbfcd8 [ 7080.237472] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7080.247587] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7080.251402] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7080.255362] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7080.260308] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7080.263869] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7080.267507] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7088.048036] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 7088.051209] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7088.073138] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7088.078012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7088.081718] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7088.083101] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7088.089917] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7088.092885] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7088.095502] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7088.097537] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7088.100034] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7088.102117] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7088.105583] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7088.110222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7088.112800] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7088.116213] Call Trace: [ 7088.118483] [ 7088.119409] ? show_regs.cold.9+0x22/0x2f [ 7088.121112] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7088.123347] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7088.127706] ? watchdog+0x40/0x40 [ 7088.129690] ? hrtimer_interrupt+0x128/0x2c0 [ 7088.131726] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7088.134233] ? apic_timer_interrupt+0xf/0x20 [ 7088.135898] [ 7088.136670] ? native_safe_halt+0x2e/0x50 [ 7088.138333] kvm_wait+0xcd/0xe0 [ 7088.139539] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7088.142373] ? dump_stack+0xf6/0x10e [ 7088.144182] do_raw_spin_lock+0x177/0x190 [ 7088.146254] _raw_spin_lock+0x21/0x30 [ 7088.148109] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7088.150853] obd_mod_free+0x46/0x140 [ptlrpc] [ 7088.153137] mdc_commit_open+0x91/0xb0 [mdc] [ 7088.154695] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7088.157273] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7088.160666] after_reply+0xe2c/0x1780 [ptlrpc] [ 7088.163141] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7088.166040] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7088.167939] ? do_wait_intr+0xf0/0xf0 [ 7088.169789] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7088.172285] kthread+0x1d1/0x200 [ 7088.173824] ? set_kthread_struct+0x70/0x70 [ 7088.176344] ret_from_fork+0x1f/0x30 [ 7092.052477] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7092.060670] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7092.085601] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7092.091081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7092.095346] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7092.098122] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7092.107567] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7092.111297] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7092.116185] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7092.119682] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7092.125375] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7092.129090] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7092.133520] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7092.137895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7092.140787] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7092.143907] Call Trace: [ 7092.145317] [ 7092.146368] ? show_regs.cold.9+0x22/0x2f [ 7092.148554] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7092.150963] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7092.153259] ? watchdog+0x40/0x40 [ 7092.155318] ? hrtimer_interrupt+0x128/0x2c0 [ 7092.160699] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7092.166036] ? apic_timer_interrupt+0xf/0x20 [ 7092.170200] [ 7092.171789] ? native_safe_halt+0x2e/0x50 [ 7092.173988] kvm_wait+0xcd/0xe0 [ 7092.175842] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7092.179302] do_raw_spin_lock+0x177/0x190 [ 7092.181834] _raw_spin_lock+0x21/0x30 [ 7092.183529] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7092.186648] ? do_raw_spin_unlock+0x75/0x190 [ 7092.189130] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7092.192523] ? do_wait_intr+0xf0/0xf0 [ 7092.194217] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7092.196950] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7092.200232] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7092.203149] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7092.205806] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7092.208644] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7092.210971] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7092.213532] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7092.216015] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7092.219608] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7092.222602] lmv_intent_open+0x338/0x1150 [lmv] [ 7092.224787] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7092.227255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7092.230275] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7092.232739] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7092.234802] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7092.239034] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7092.240794] ? do_raw_spin_unlock+0x75/0x190 [ 7092.242029] ? _raw_spin_unlock+0x12/0x30 [ 7092.243661] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7092.245005] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7092.247376] ll_atomic_open+0x36a/0xd70 [lustre] [ 7092.250637] lookup_open+0x356/0xb40 [ 7092.253471] do_last+0x313/0xfc0 [ 7092.254649] ? nd_jump_root+0xe5/0x160 [ 7092.259492] ? path_init+0x437/0x520 [ 7092.262490] path_openat+0xf7/0x500 [ 7092.264544] ? do_raw_spin_unlock+0x75/0x190 [ 7092.268049] do_filp_open+0x99/0x140 [ 7092.271128] ? getname_flags+0x6e/0x330 [ 7092.273281] ? __check_object_size+0xff/0x256 [ 7092.276076] ? do_raw_spin_unlock+0x75/0x190 [ 7092.278966] ? _raw_spin_unlock+0x12/0x30 [ 7092.280713] do_sys_openat2+0x2b4/0x410 [ 7092.282498] do_sys_open+0x73/0xa0 [ 7092.283776] __x64_sys_openat+0x24/0x30 [ 7092.285804] do_syscall_64+0xc1/0x440 [ 7092.288524] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7092.290892] RIP: 0033:0x7fdc0afbf332 [ 7092.292763] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7092.303412] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7092.308528] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7092.312513] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7092.316065] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7092.322679] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7092.326513] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7092.703193] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5067s! [ 7092.706493] Showing busy workqueues and worker pools: [ 7092.708722] workqueue events: flags=0x0 [ 7092.710638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7092.713797] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7092.717551] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7092.720600] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7092.725381] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7092.727257] pending: kfree_rcu_monitor [ 7092.728571] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7092.731751] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7092.738142] workqueue events_long: flags=0x0 [ 7092.740037] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7092.743565] pending: lu_objects_flush [obdclass] [ 7092.746415] workqueue events_unbound: flags=0x2 [ 7092.748490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7092.751265] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7092.755894] workqueue events_power_efficient: flags=0x80 [ 7092.758801] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7092.762331] pending: neigh_periodic_work, check_lifetime [ 7092.764621] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7092.767445] pending: do_cache_clean, neigh_periodic_work [ 7092.770511] workqueue mm_percpu_wq: flags=0x8 [ 7092.772603] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7092.775656] pending: lru_add_drain_per_cpu [ 7092.777530] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7092.780584] pending: vmstat_update, lru_add_drain_per_cpu [ 7092.782939] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7092.786073] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7092.788882] workqueue writeback: flags=0x4a [ 7092.790352] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7092.792504] pending: wb_workfn, wb_workfn [ 7092.795478] workqueue xprtiod: flags=0xa [ 7092.797013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7092.800128] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7092.803924] workqueue ipv6_addrconf: flags=0x40008 [ 7092.807888] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7092.811049] pending: addrconf_verify_work [ 7092.814250] workqueue knbd0-recv: flags=0x1a [ 7092.816176] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7092.819499] in-flight: 655:recv_work [ 7092.821373] workqueue ptlrpc_pinger: flags=0x2 [ 7092.823690] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7092.828757] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7092.832531] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5067s workers=4 idle: 36396 38 11188 [ 7092.836291] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7083s workers=2 idle: 658 [ 7104.051023] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7104.055155] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7104.086060] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7104.092681] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7104.097700] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7104.101038] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7104.103287] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7104.113222] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7104.116969] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7104.120459] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7104.124009] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7104.127547] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7104.131330] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7104.136515] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7104.143291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7104.147957] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7104.153192] Call Trace: [ 7104.157283] [ 7104.159179] ? show_regs.cold.9+0x22/0x2f [ 7104.163579] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7104.166858] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7104.172968] ? watchdog+0x40/0x40 [ 7104.175857] ? hrtimer_interrupt+0x128/0x2c0 [ 7104.181037] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7104.184488] ? apic_timer_interrupt+0xf/0x20 [ 7104.187827] [ 7104.189324] ? native_safe_halt+0x2e/0x50 [ 7104.193338] kvm_wait+0xcd/0xe0 [ 7104.196555] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7104.200330] do_raw_spin_lock+0x177/0x190 [ 7104.202153] _raw_spin_lock+0x21/0x30 [ 7104.204500] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7104.206284] process_one_work+0x2c8/0x700 [ 7104.207834] worker_thread+0x296/0x6e0 [ 7104.209333] ? rescuer_thread+0x570/0x570 [ 7104.211243] kthread+0x1d1/0x200 [ 7104.212528] ? set_kthread_struct+0x70/0x70 [ 7104.214225] ret_from_fork+0x1f/0x30 [ 7108.054071] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7108.057297] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7108.071443] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7108.074758] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7108.077715] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7108.079573] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7108.086183] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7108.088464] RAX: 00000000000011d1 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7108.090861] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7108.093184] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7108.095705] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7108.098296] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7108.100388] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7108.102748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7108.104571] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7108.106822] Call Trace: [ 7108.107387] [ 7108.107864] ? show_regs.cold.9+0x22/0x2f [ 7108.109256] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7108.110445] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7108.111395] ? watchdog+0x40/0x40 [ 7108.112206] ? hrtimer_interrupt+0x128/0x2c0 [ 7108.113125] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7108.113997] ? apic_timer_interrupt+0xf/0x20 [ 7108.115526] [ 7108.116236] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7108.117698] do_raw_spin_lock+0x177/0x190 [ 7108.118606] _raw_spin_lock+0x21/0x30 [ 7108.119383] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7108.120802] ? do_raw_spin_unlock+0x75/0x190 [ 7108.122128] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7108.123861] ? do_wait_intr+0xf0/0xf0 [ 7108.125080] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7108.126940] mdc_close+0x53b/0xd70 [mdc] [ 7108.128238] lmv_close+0x18d/0x490 [lmv] [ 7108.128974] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7108.130367] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7108.131453] ll_md_close+0x274/0x800 [lustre] [ 7108.132675] ll_file_release+0x128/0xc50 [lustre] [ 7108.134599] __fput+0x10d/0x440 [ 7108.135632] ____fput+0x12/0x20 [ 7108.136549] task_work_run+0xc6/0x110 [ 7108.137531] exit_to_usermode_loop+0x1dd/0x1f0 [ 7108.138678] do_syscall_64+0x3d6/0x440 [ 7108.139638] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7108.141126] RIP: 0033:0x7fdc0afbfcd8 [ 7108.142146] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7108.147901] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7108.150316] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7108.152366] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7108.154388] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7108.156630] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7108.158780] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7119.455126] rcu: INFO: rcu_sched self-detected stall on CPU [ 7119.458991] rcu: 0-...!: (3115982 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 7119.466153] (t=5100112 jiffies g=270761 q=3888) [ 7119.468461] rcu: rcu_sched kthread starved for 5095044 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 7119.475137] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7119.479159] rcu: RCU grace-period kthread stack dump: [ 7119.481841] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7119.486653] Call Trace: [ 7119.487745] __schedule+0x351/0xcb0 [ 7119.489477] schedule+0xc0/0x180 [ 7119.491116] schedule_timeout+0xb4/0x190 [ 7119.493004] ? do_raw_spin_unlock+0x75/0x190 [ 7119.495012] ? __next_timer_interrupt+0x160/0x160 [ 7119.497304] ? prepare_to_swait_event+0xf3/0x140 [ 7119.500658] rcu_gp_kthread+0x8dd/0x15b0 [ 7119.503001] ? force_qs_rnp+0x2c0/0x2c0 [ 7119.506441] kthread+0x1d1/0x200 [ 7119.508088] ? set_kthread_struct+0x70/0x70 [ 7119.511352] ret_from_fork+0x1f/0x30 [ 7119.513111] rcu: Stack dump where RCU GP kthread last ran: [ 7119.515569] NMI backtrace for cpu 0 [ 7119.517151] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7119.522956] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7119.527174] Call Trace: [ 7119.528373] [ 7119.529720] ? dump_stack+0xbb/0x10e [ 7119.531781] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7119.534118] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7119.536531] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7119.539269] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7119.542883] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7119.545271] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 7119.547931] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 7119.550120] ? update_process_times+0x6d/0xc0 [ 7119.551938] ? tick_sched_handle.isra.18+0x27/0x90 [ 7119.554669] ? tick_sched_timer+0x53/0xb0 [ 7119.556255] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7119.558321] ? tick_sched_handle.isra.18+0x90/0x90 [ 7119.561538] ? hrtimer_interrupt+0x128/0x2c0 [ 7119.563783] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7119.567200] ? apic_timer_interrupt+0xf/0x20 [ 7119.570109] [ 7119.571142] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 7119.575969] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7119.581093] ? dump_stack+0xf6/0x10e [ 7119.583107] ? do_raw_spin_lock+0x177/0x190 [ 7119.585129] ? _raw_spin_lock+0x21/0x30 [ 7119.586950] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7119.589912] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7119.592367] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7119.594945] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7119.598712] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7119.607358] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7119.611327] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7119.614704] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7119.617949] ? do_wait_intr+0xf0/0xf0 [ 7119.619908] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7119.622496] ? kthread+0x1d1/0x200 [ 7119.624505] ? set_kthread_struct+0x70/0x70 [ 7119.626426] ? ret_from_fork+0x1f/0x30 [ 7119.627764] NMI backtrace for cpu 0 [ 7119.628905] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7119.632134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7119.634970] Call Trace: [ 7119.635578] [ 7119.636054] ? dump_stack+0xbb/0x10e [ 7119.636860] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7119.638439] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7119.639690] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7119.641143] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7119.642655] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7119.644267] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7119.645973] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7119.647579] ? update_process_times+0x6d/0xc0 [ 7119.649166] ? tick_sched_handle.isra.18+0x27/0x90 [ 7119.650570] ? tick_sched_timer+0x53/0xb0 [ 7119.651651] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7119.653631] ? tick_sched_handle.isra.18+0x90/0x90 [ 7119.655341] ? hrtimer_interrupt+0x128/0x2c0 [ 7119.657317] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7119.659287] ? apic_timer_interrupt+0xf/0x20 [ 7119.660608] [ 7119.661761] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 7119.663522] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7119.665638] ? dump_stack+0xf6/0x10e [ 7119.667438] ? do_raw_spin_lock+0x177/0x190 [ 7119.669458] ? _raw_spin_lock+0x21/0x30 [ 7119.671097] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7119.673530] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7119.675769] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7119.678263] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7119.680429] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7119.682613] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7119.684481] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7119.686550] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7119.688326] ? do_wait_intr+0xf0/0xf0 [ 7119.689886] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7119.691800] ? kthread+0x1d1/0x200 [ 7119.692973] ? set_kthread_struct+0x70/0x70 [ 7119.694523] ? ret_from_fork+0x1f/0x30 [ 7120.052024] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 7120.055185] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7120.084648] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7120.089751] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7120.094486] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7120.099288] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7120.106106] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7120.116492] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7120.119747] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7120.123117] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7120.131304] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7120.134492] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7120.141273] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7120.146128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7120.149390] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7120.155554] Call Trace: [ 7120.157926] [ 7120.159315] ? show_regs.cold.9+0x22/0x2f [ 7120.162295] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7120.164551] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7120.167482] ? watchdog+0x40/0x40 [ 7120.169559] ? hrtimer_interrupt+0x128/0x2c0 [ 7120.171891] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7120.174917] ? apic_timer_interrupt+0xf/0x20 [ 7120.180612] [ 7120.182397] ? native_safe_halt+0x2e/0x50 [ 7120.183687] kvm_wait+0xcd/0xe0 [ 7120.184628] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7120.188896] do_raw_spin_lock+0x177/0x190 [ 7120.192582] _raw_spin_lock+0x21/0x30 [ 7120.194040] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7120.196677] ? do_raw_spin_unlock+0x75/0x190 [ 7120.198519] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7120.200152] ? do_wait_intr+0xf0/0xf0 [ 7120.201999] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7120.204383] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7120.206554] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7120.208920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7120.211346] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7120.212781] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7120.214405] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7120.215718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7120.218095] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7120.220602] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7120.222981] lmv_intent_open+0x338/0x1150 [lmv] [ 7120.224849] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7120.228866] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7120.231476] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7120.237236] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7120.239424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7120.244798] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7120.246490] ? do_raw_spin_unlock+0x75/0x190 [ 7120.247901] ? _raw_spin_unlock+0x12/0x30 [ 7120.252459] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7120.257289] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7120.258943] ll_atomic_open+0x36a/0xd70 [lustre] [ 7120.261232] lookup_open+0x356/0xb40 [ 7120.262642] do_last+0x313/0xfc0 [ 7120.264687] ? nd_jump_root+0xe5/0x160 [ 7120.266323] ? path_init+0x437/0x520 [ 7120.270771] path_openat+0xf7/0x500 [ 7120.273499] ? do_raw_spin_unlock+0x75/0x190 [ 7120.276613] do_filp_open+0x99/0x140 [ 7120.278424] ? getname_flags+0x6e/0x330 [ 7120.280381] ? __check_object_size+0xff/0x256 [ 7120.286399] ? do_raw_spin_unlock+0x75/0x190 [ 7120.291846] ? _raw_spin_unlock+0x12/0x30 [ 7120.292921] do_sys_openat2+0x2b4/0x410 [ 7120.293959] do_sys_open+0x73/0xa0 [ 7120.294855] __x64_sys_openat+0x24/0x30 [ 7120.304738] do_syscall_64+0xc1/0x440 [ 7120.306054] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7120.312107] RIP: 0033:0x7fdc0afbf332 [ 7120.313660] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7120.323032] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7120.329937] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7120.339162] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7120.343893] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7120.351055] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7120.354186] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7122.911908] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5097s! [ 7122.925700] Showing busy workqueues and worker pools: [ 7122.933257] workqueue events: flags=0x0 [ 7122.934873] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7122.937144] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7122.940428] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7122.944084] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7122.952886] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7122.956103] pending: kfree_rcu_monitor [ 7122.959706] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7122.963630] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7122.981642] workqueue events_long: flags=0x0 [ 7122.984968] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7122.991203] pending: lu_objects_flush [obdclass] [ 7122.993129] workqueue events_unbound: flags=0x2 [ 7123.002385] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7123.004540] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7123.009669] workqueue events_power_efficient: flags=0x80 [ 7123.011725] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7123.014366] pending: neigh_periodic_work, check_lifetime [ 7123.016376] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7123.019803] pending: do_cache_clean, neigh_periodic_work [ 7123.022435] workqueue mm_percpu_wq: flags=0x8 [ 7123.024854] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7123.028337] pending: lru_add_drain_per_cpu [ 7123.030358] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7123.032724] pending: vmstat_update, lru_add_drain_per_cpu [ 7123.034878] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7123.037118] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7123.040256] workqueue writeback: flags=0x4a [ 7123.041984] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7123.045217] pending: wb_workfn, wb_workfn [ 7123.047337] workqueue xprtiod: flags=0xa [ 7123.049255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7123.053670] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7123.057511] workqueue ipv6_addrconf: flags=0x40008 [ 7123.060199] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7123.063132] pending: addrconf_verify_work [ 7123.065183] workqueue knbd0-recv: flags=0x1a [ 7123.066729] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7123.070038] in-flight: 655:recv_work [ 7123.072356] workqueue ptlrpc_pinger: flags=0x2 [ 7123.074426] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7123.077098] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7123.079875] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5098s workers=4 idle: 36396 38 11188 [ 7123.084144] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7113s workers=2 idle: 658 [ 7132.051013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7132.053838] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7132.072315] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7132.076378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7132.080182] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7132.083332] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7132.085979] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7132.094049] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7132.097547] RAX: 0000000000003905 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7132.100673] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7132.103809] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7132.106852] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7132.109963] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7132.113256] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7132.116696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7132.119431] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7132.123087] Call Trace: [ 7132.124204] [ 7132.125215] ? show_regs.cold.9+0x22/0x2f [ 7132.127359] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7132.130150] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7132.131788] ? watchdog+0x40/0x40 [ 7132.132950] ? hrtimer_interrupt+0x128/0x2c0 [ 7132.135211] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7132.137442] ? apic_timer_interrupt+0xf/0x20 [ 7132.139438] [ 7132.140425] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7132.142858] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7132.145185] do_raw_spin_lock+0x177/0x190 [ 7132.147246] _raw_spin_lock+0x21/0x30 [ 7132.148984] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7132.151518] process_one_work+0x2c8/0x700 [ 7132.153509] worker_thread+0x296/0x6e0 [ 7132.155219] ? rescuer_thread+0x570/0x570 [ 7132.156801] kthread+0x1d1/0x200 [ 7132.158218] ? set_kthread_struct+0x70/0x70 [ 7132.159985] ret_from_fork+0x1f/0x30 [ 7136.054070] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7136.057784] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7136.080688] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7136.096692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7136.105008] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7136.107968] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7136.118632] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7136.122401] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7136.126548] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7136.130228] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7136.134099] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7136.137767] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7136.141432] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7136.145630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7136.149250] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7136.153782] Call Trace: [ 7136.155078] [ 7136.156844] ? show_regs.cold.9+0x22/0x2f [ 7136.158854] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7136.161448] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7136.163653] ? watchdog+0x40/0x40 [ 7136.166091] ? hrtimer_interrupt+0x128/0x2c0 [ 7136.169056] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7136.173622] ? apic_timer_interrupt+0xf/0x20 [ 7136.176135] [ 7136.177657] ? native_safe_halt+0x2e/0x50 [ 7136.181422] kvm_wait+0xcd/0xe0 [ 7136.184620] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7136.188655] do_raw_spin_lock+0x177/0x190 [ 7136.192450] _raw_spin_lock+0x21/0x30 [ 7136.195644] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7136.200496] ? do_raw_spin_unlock+0x75/0x190 [ 7136.202947] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7136.208290] ? do_wait_intr+0xf0/0xf0 [ 7136.210603] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7136.213658] mdc_close+0x53b/0xd70 [mdc] [ 7136.217025] lmv_close+0x18d/0x490 [lmv] [ 7136.219347] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7136.222293] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7136.224057] ll_md_close+0x274/0x800 [lustre] [ 7136.225753] ll_file_release+0x128/0xc50 [lustre] [ 7136.229118] __fput+0x10d/0x440 [ 7136.230248] ____fput+0x12/0x20 [ 7136.231238] task_work_run+0xc6/0x110 [ 7136.232612] exit_to_usermode_loop+0x1dd/0x1f0 [ 7136.234952] do_syscall_64+0x3d6/0x440 [ 7136.236938] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7136.239478] RIP: 0033:0x7fdc0afbfcd8 [ 7136.241355] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7136.253273] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7136.258318] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7136.262354] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7136.265096] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7136.267737] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7136.270336] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7148.048023] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 7148.052095] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7148.052758] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 7148.074211] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7148.074227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7148.074231] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7148.074246] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7148.074250] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7148.074256] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7148.074258] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7148.074261] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7148.074263] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7148.074265] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7148.074269] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7148.074272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7148.074274] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7148.074282] Call Trace: [ 7148.074288] [ 7148.074294] ? show_regs.cold.9+0x22/0x2f [ 7148.074317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7148.074324] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7148.074328] ? watchdog+0x40/0x40 [ 7148.074333] ? hrtimer_interrupt+0x128/0x2c0 [ 7148.074337] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7148.074345] ? apic_timer_interrupt+0xf/0x20 [ 7148.074349] [ 7148.074353] ? native_safe_halt+0x2e/0x50 [ 7148.074357] kvm_wait+0xcd/0xe0 [ 7148.074365] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7148.074371] ? dump_stack+0xf6/0x10e [ 7148.074377] do_raw_spin_lock+0x177/0x190 [ 7148.074381] _raw_spin_lock+0x21/0x30 [ 7148.074388] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7148.076691] Modules linked in: [ 7148.081382] obd_mod_free+0x46/0x140 [ptlrpc] [ 7148.084558] lustre(O) [ 7148.086784] mdc_commit_open+0x91/0xb0 [mdc] [ 7148.086836] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7148.087190] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7148.087400] after_reply+0xe2c/0x1780 [ptlrpc] [ 7148.109516] osp(O) [ 7148.113171] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7148.116537] ofd(O) [ 7148.119175] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7148.119477] ? do_wait_intr+0xf0/0xf0 [ 7148.119488] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7148.119706] kthread+0x1d1/0x200 [ 7148.119727] ? set_kthread_struct+0x70/0x70 [ 7148.119732] ret_from_fork+0x1f/0x30 [ 7148.261098] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7148.290128] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7148.295739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7148.299444] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7148.301285] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7148.309565] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7148.312814] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7148.315731] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7148.318473] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7148.321269] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7148.324478] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7148.327180] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7148.330821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7148.333469] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7148.336443] Call Trace: [ 7148.337465] [ 7148.338412] ? show_regs.cold.9+0x22/0x2f [ 7148.340304] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7148.342176] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7148.344061] ? watchdog+0x40/0x40 [ 7148.345420] ? hrtimer_interrupt+0x128/0x2c0 [ 7148.347010] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7148.349161] ? apic_timer_interrupt+0xf/0x20 [ 7148.350869] [ 7148.351691] ? native_safe_halt+0x2e/0x50 [ 7148.353369] kvm_wait+0xcd/0xe0 [ 7148.354668] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7148.359815] do_raw_spin_lock+0x177/0x190 [ 7148.361186] _raw_spin_lock+0x21/0x30 [ 7148.364034] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7148.368538] ? do_raw_spin_unlock+0x75/0x190 [ 7148.373382] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7148.375630] ? do_wait_intr+0xf0/0xf0 [ 7148.379877] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7148.383488] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7148.388235] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7148.393884] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7148.397095] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7148.401748] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7148.404445] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7148.408431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7148.414184] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7148.421722] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7148.429058] lmv_intent_open+0x338/0x1150 [lmv] [ 7148.431876] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7148.437861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7148.442724] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7148.447604] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7148.449544] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7148.455171] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7148.460101] ? do_raw_spin_unlock+0x75/0x190 [ 7148.462182] ? _raw_spin_unlock+0x12/0x30 [ 7148.465248] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7148.470224] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7148.475731] ll_atomic_open+0x36a/0xd70 [lustre] [ 7148.480053] lookup_open+0x356/0xb40 [ 7148.483821] do_last+0x313/0xfc0 [ 7148.487051] ? nd_jump_root+0xe5/0x160 [ 7148.490356] ? path_init+0x437/0x520 [ 7148.494743] path_openat+0xf7/0x500 [ 7148.500640] ? do_raw_spin_unlock+0x75/0x190 [ 7148.503847] do_filp_open+0x99/0x140 [ 7148.507229] ? getname_flags+0x6e/0x330 [ 7148.513578] ? __check_object_size+0xff/0x256 [ 7148.520125] ? do_raw_spin_unlock+0x75/0x190 [ 7148.526135] ? _raw_spin_unlock+0x12/0x30 [ 7148.532317] do_sys_openat2+0x2b4/0x410 [ 7148.537122] do_sys_open+0x73/0xa0 [ 7148.542197] __x64_sys_openat+0x24/0x30 [ 7148.546783] do_syscall_64+0xc1/0x440 [ 7148.550763] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7148.554489] RIP: 0033:0x7fdc0afbf332 [ 7148.558077] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7148.576573] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7148.583976] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7148.587398] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7148.593371] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7148.597712] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7148.602856] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7153.119045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5128s! [ 7153.123411] Showing busy workqueues and worker pools: [ 7153.125936] workqueue events: flags=0x0 [ 7153.128100] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7153.131856] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7153.135424] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7153.138976] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7153.144922] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7153.148168] pending: kfree_rcu_monitor [ 7153.150557] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7153.153792] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7153.160157] workqueue events_long: flags=0x0 [ 7153.162611] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7153.165913] pending: lu_objects_flush [obdclass] [ 7153.168810] workqueue events_unbound: flags=0x2 [ 7153.171798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7153.174711] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7153.180582] workqueue events_power_efficient: flags=0x80 [ 7153.183211] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7153.186851] pending: neigh_periodic_work, check_lifetime [ 7153.190242] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7153.193762] pending: do_cache_clean, neigh_periodic_work [ 7153.196662] workqueue mm_percpu_wq: flags=0x8 [ 7153.198676] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7153.203746] pending: lru_add_drain_per_cpu [ 7153.206402] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7153.210075] pending: vmstat_update, lru_add_drain_per_cpu [ 7153.213718] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7153.216763] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7153.219967] workqueue writeback: flags=0x4a [ 7153.222030] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7153.225706] pending: wb_workfn, wb_workfn [ 7153.227931] workqueue xprtiod: flags=0xa [ 7153.229910] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7153.232965] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7153.236840] workqueue ipv6_addrconf: flags=0x40008 [ 7153.239239] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7153.242776] pending: addrconf_verify_work [ 7153.245282] workqueue knbd0-recv: flags=0x1a [ 7153.247393] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7153.251109] in-flight: 655:recv_work [ 7153.253311] workqueue ptlrpc_pinger: flags=0x2 [ 7153.255185] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7153.258349] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7153.262338] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5128s workers=4 idle: 36396 38 11188 [ 7153.265977] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7143s workers=2 idle: 658 [ 7160.051255] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7160.053670] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7160.074102] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7160.078627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7160.081916] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7160.084691] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7160.086833] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7160.103691] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7160.109060] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7160.114015] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7160.116922] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7160.123474] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7160.127143] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7160.132887] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7160.136437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7160.142481] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7160.152427] Call Trace: [ 7160.153783] [ 7160.154360] ? show_regs.cold.9+0x22/0x2f [ 7160.155547] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7160.163035] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7160.164913] ? watchdog+0x40/0x40 [ 7160.165960] ? hrtimer_interrupt+0x128/0x2c0 [ 7160.169027] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7160.170336] ? apic_timer_interrupt+0xf/0x20 [ 7160.171411] [ 7160.173576] ? native_safe_halt+0x2e/0x50 [ 7160.178749] kvm_wait+0xcd/0xe0 [ 7160.181595] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7160.183805] do_raw_spin_lock+0x177/0x190 [ 7160.185623] _raw_spin_lock+0x21/0x30 [ 7160.186878] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7160.189601] process_one_work+0x2c8/0x700 [ 7160.190867] worker_thread+0x296/0x6e0 [ 7160.191899] ? rescuer_thread+0x570/0x570 [ 7160.194880] kthread+0x1d1/0x200 [ 7160.196131] ? set_kthread_struct+0x70/0x70 [ 7160.199085] ret_from_fork+0x1f/0x30 [ 7164.059545] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 7164.066859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7164.107014] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7164.112532] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7164.120602] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7164.125900] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7164.143872] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7164.146927] RAX: 0000000000006b21 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7164.151258] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7164.154999] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7164.157954] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7164.161869] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7164.169385] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7164.174291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7164.176089] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7164.179850] Call Trace: [ 7164.181536] [ 7164.182420] ? show_regs.cold.9+0x22/0x2f [ 7164.184397] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7164.187011] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7164.188909] ? watchdog+0x40/0x40 [ 7164.190981] ? hrtimer_interrupt+0x128/0x2c0 [ 7164.196831] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7164.198768] ? apic_timer_interrupt+0xf/0x20 [ 7164.205473] [ 7164.207998] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7164.212092] do_raw_spin_lock+0x177/0x190 [ 7164.214254] _raw_spin_lock+0x21/0x30 [ 7164.216299] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7164.222203] ? do_raw_spin_unlock+0x75/0x190 [ 7164.225455] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7164.231005] ? do_wait_intr+0xf0/0xf0 [ 7164.232914] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7164.235708] mdc_close+0x53b/0xd70 [mdc] [ 7164.237944] lmv_close+0x18d/0x490 [lmv] [ 7164.239768] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7164.243194] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7164.245910] ll_md_close+0x274/0x800 [lustre] [ 7164.250224] ll_file_release+0x128/0xc50 [lustre] [ 7164.252884] __fput+0x10d/0x440 [ 7164.258840] ____fput+0x12/0x20 [ 7164.265397] task_work_run+0xc6/0x110 [ 7164.268805] exit_to_usermode_loop+0x1dd/0x1f0 [ 7164.274633] do_syscall_64+0x3d6/0x440 [ 7164.279306] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7164.282751] RIP: 0033:0x7fdc0afbfcd8 [ 7164.285369] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7164.298405] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7164.303857] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7164.307663] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7164.312663] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7164.316193] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7164.327239] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7176.052035] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 7176.056215] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7176.076429] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7176.081339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7176.085364] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7176.088370] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7176.096486] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7176.100742] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7176.104712] RDX: 00000000000047c6 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7176.108334] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7176.112290] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7176.116349] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7176.120196] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7176.125069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7176.128649] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7176.132418] Call Trace: [ 7176.133496] [ 7176.134640] ? show_regs.cold.9+0x22/0x2f [ 7176.137084] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7176.139552] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7176.142237] ? watchdog+0x40/0x40 [ 7176.143992] ? hrtimer_interrupt+0x128/0x2c0 [ 7176.146139] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7176.148881] ? apic_timer_interrupt+0xf/0x20 [ 7176.151696] [ 7176.152988] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7176.155608] do_raw_spin_lock+0x177/0x190 [ 7176.157237] _raw_spin_lock+0x21/0x30 [ 7176.158796] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7176.161508] ? do_raw_spin_unlock+0x75/0x190 [ 7176.163783] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7176.166656] ? do_wait_intr+0xf0/0xf0 [ 7176.168189] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7176.170278] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7176.173075] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7176.176588] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7176.179655] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7176.181991] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7176.184048] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7176.186542] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7176.189588] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7176.192681] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7176.195155] lmv_intent_open+0x338/0x1150 [lmv] [ 7176.197269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7176.199924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7176.201672] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7176.203186] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7176.205228] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7176.207506] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7176.209648] ? do_raw_spin_unlock+0x75/0x190 [ 7176.211626] ? _raw_spin_unlock+0x12/0x30 [ 7176.213271] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7176.215856] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7176.218088] ll_atomic_open+0x36a/0xd70 [lustre] [ 7176.220351] lookup_open+0x356/0xb40 [ 7176.222025] do_last+0x313/0xfc0 [ 7176.223933] ? nd_jump_root+0xe5/0x160 [ 7176.225238] ? path_init+0x437/0x520 [ 7176.228016] path_openat+0xf7/0x500 [ 7176.229627] ? do_raw_spin_unlock+0x75/0x190 [ 7176.236608] do_filp_open+0x99/0x140 [ 7176.238370] ? getname_flags+0x6e/0x330 [ 7176.246516] ? __check_object_size+0xff/0x256 [ 7176.256075] ? do_raw_spin_unlock+0x75/0x190 [ 7176.258793] ? _raw_spin_unlock+0x12/0x30 [ 7176.261108] do_sys_openat2+0x2b4/0x410 [ 7176.268197] do_sys_open+0x73/0xa0 [ 7176.270125] __x64_sys_openat+0x24/0x30 [ 7176.273753] do_syscall_64+0xc1/0x440 [ 7176.275555] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7176.278595] RIP: 0033:0x7fdc0afbf332 [ 7176.280344] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7176.289876] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7176.297166] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7176.305164] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7176.312598] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7176.318601] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7176.324282] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7180.048055] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7180.050923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7180.079861] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7180.087511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7180.095110] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7180.099570] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7180.106567] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7180.109720] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7180.112951] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7180.117356] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7180.121361] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7180.128942] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7180.135712] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7180.146455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7180.155108] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7180.158683] Call Trace: [ 7180.160182] [ 7180.161929] ? show_regs.cold.9+0x22/0x2f [ 7180.166252] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7180.169283] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7180.171093] ? watchdog+0x40/0x40 [ 7180.174466] ? hrtimer_interrupt+0x128/0x2c0 [ 7180.177112] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7180.179904] ? apic_timer_interrupt+0xf/0x20 [ 7180.181823] [ 7180.184432] ? native_safe_halt+0x2e/0x50 [ 7180.188250] kvm_wait+0xcd/0xe0 [ 7180.190681] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7180.197696] ? dump_stack+0xf6/0x10e [ 7180.200980] do_raw_spin_lock+0x177/0x190 [ 7180.204921] _raw_spin_lock+0x21/0x30 [ 7180.207599] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7180.217950] obd_mod_free+0x46/0x140 [ptlrpc] [ 7180.225870] mdc_commit_open+0x91/0xb0 [mdc] [ 7180.230020] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7180.233358] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7180.238136] after_reply+0xe2c/0x1780 [ptlrpc] [ 7180.242623] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7180.246798] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7180.250724] ? do_wait_intr+0xf0/0xf0 [ 7180.253277] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7180.258832] kthread+0x1d1/0x200 [ 7180.261575] ? set_kthread_struct+0x70/0x70 [ 7180.265634] ret_from_fork+0x1f/0x30 [ 7183.327048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5158s! [ 7183.330168] Showing busy workqueues and worker pools: [ 7183.332612] workqueue events: flags=0x0 [ 7183.333970] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7183.336786] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7183.338878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7183.340786] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7183.343860] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7183.345866] pending: kfree_rcu_monitor [ 7183.347202] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7183.349488] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7183.353435] workqueue events_long: flags=0x0 [ 7183.354789] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7183.356737] pending: lu_objects_flush [obdclass] [ 7183.358272] workqueue events_unbound: flags=0x2 [ 7183.360117] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7183.362191] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7183.365646] workqueue events_power_efficient: flags=0x80 [ 7183.367501] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7183.369876] pending: neigh_periodic_work, check_lifetime [ 7183.372669] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7183.375982] pending: do_cache_clean, neigh_periodic_work [ 7183.380268] workqueue mm_percpu_wq: flags=0x8 [ 7183.382922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7183.385071] pending: lru_add_drain_per_cpu [ 7183.386820] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7183.388867] pending: vmstat_update, lru_add_drain_per_cpu [ 7183.390830] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7183.392712] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7183.394495] workqueue writeback: flags=0x4a [ 7183.395701] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7183.397527] pending: wb_workfn, wb_workfn [ 7183.398764] workqueue xprtiod: flags=0xa [ 7183.399907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7183.401599] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7183.403983] workqueue ipv6_addrconf: flags=0x40008 [ 7183.405448] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7183.407328] pending: addrconf_verify_work [ 7183.408550] workqueue knbd0-recv: flags=0x1a [ 7183.409953] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7183.412677] in-flight: 655:recv_work [ 7183.414268] workqueue ptlrpc_pinger: flags=0x2 [ 7183.415863] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7183.418223] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7183.421414] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5158s workers=4 idle: 36396 38 11188 [ 7183.424571] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7173s workers=2 idle: 658 [ 7188.051036] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7188.055503] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7188.076417] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7188.080677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7188.085717] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7188.088093] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7188.089947] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7188.097730] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7188.100255] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7188.102331] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7188.104572] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7188.107205] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7188.111193] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7188.116181] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7188.120577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7188.123368] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7188.128287] Call Trace: [ 7188.129200] [ 7188.130418] ? show_regs.cold.9+0x22/0x2f [ 7188.131868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7188.133175] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7188.134497] ? watchdog+0x40/0x40 [ 7188.136028] ? hrtimer_interrupt+0x128/0x2c0 [ 7188.137286] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7188.140048] ? apic_timer_interrupt+0xf/0x20 [ 7188.141279] [ 7188.142211] ? native_safe_halt+0x2e/0x50 [ 7188.144680] kvm_wait+0xcd/0xe0 [ 7188.145915] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7188.147892] do_raw_spin_lock+0x177/0x190 [ 7188.151254] _raw_spin_lock+0x21/0x30 [ 7188.152354] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7188.155296] process_one_work+0x2c8/0x700 [ 7188.156849] worker_thread+0x296/0x6e0 [ 7188.158438] ? rescuer_thread+0x570/0x570 [ 7188.162414] kthread+0x1d1/0x200 [ 7188.163700] ? set_kthread_struct+0x70/0x70 [ 7188.166596] ret_from_fork+0x1f/0x30 [ 7192.054021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 7192.059410] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7192.085935] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7192.090684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7192.098473] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7192.100208] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7192.110170] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7192.116637] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7192.121725] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7192.129447] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7192.134005] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7192.136729] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7192.141808] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7192.145081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7192.147321] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7192.152985] Call Trace: [ 7192.153641] [ 7192.154198] ? show_regs.cold.9+0x22/0x2f [ 7192.155562] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7192.157032] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7192.158317] ? watchdog+0x40/0x40 [ 7192.159292] ? hrtimer_interrupt+0x128/0x2c0 [ 7192.160503] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7192.165653] ? apic_timer_interrupt+0xf/0x20 [ 7192.167336] [ 7192.168312] ? native_safe_halt+0x2e/0x50 [ 7192.170049] kvm_wait+0xcd/0xe0 [ 7192.172875] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7192.174273] do_raw_spin_lock+0x177/0x190 [ 7192.175405] _raw_spin_lock+0x21/0x30 [ 7192.176900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7192.178973] ? do_raw_spin_unlock+0x75/0x190 [ 7192.182010] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7192.184226] ? do_wait_intr+0xf0/0xf0 [ 7192.186361] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7192.188345] mdc_close+0x53b/0xd70 [mdc] [ 7192.189772] lmv_close+0x18d/0x490 [lmv] [ 7192.191324] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7192.193759] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7192.196227] ll_md_close+0x274/0x800 [lustre] [ 7192.199094] ll_file_release+0x128/0xc50 [lustre] [ 7192.202079] __fput+0x10d/0x440 [ 7192.203767] ____fput+0x12/0x20 [ 7192.204970] task_work_run+0xc6/0x110 [ 7192.206452] exit_to_usermode_loop+0x1dd/0x1f0 [ 7192.208469] do_syscall_64+0x3d6/0x440 [ 7192.209842] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7192.211943] RIP: 0033:0x7fdc0afbfcd8 [ 7192.213453] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7192.220047] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7192.222641] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7192.225030] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7192.227298] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7192.230033] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7192.232933] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7204.052023] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 7204.057123] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7204.085562] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7204.090910] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7204.099985] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7204.103740] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7204.113698] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7204.118509] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7204.122973] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7204.124565] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7204.128271] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7204.135654] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7204.142275] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7204.150713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7204.156750] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7204.160299] Call Trace: [ 7204.162925] [ 7204.163578] ? show_regs.cold.9+0x22/0x2f [ 7204.166186] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7204.169873] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7204.172616] ? watchdog+0x40/0x40 [ 7204.173788] ? hrtimer_interrupt+0x128/0x2c0 [ 7204.176086] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7204.179112] ? apic_timer_interrupt+0xf/0x20 [ 7204.183421] [ 7204.184807] ? native_safe_halt+0x2e/0x50 [ 7204.188214] kvm_wait+0xcd/0xe0 [ 7204.194421] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7204.198293] do_raw_spin_lock+0x177/0x190 [ 7204.200873] _raw_spin_lock+0x21/0x30 [ 7204.203988] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7204.208909] ? do_raw_spin_unlock+0x75/0x190 [ 7204.214890] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7204.220811] ? do_wait_intr+0xf0/0xf0 [ 7204.226444] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7204.236520] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7204.244808] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7204.254326] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7204.260302] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7204.263219] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7204.266967] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7204.270888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7204.277223] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7204.280702] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7204.284264] lmv_intent_open+0x338/0x1150 [lmv] [ 7204.286923] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7204.290086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7204.293186] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7204.295478] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7204.298860] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7204.304305] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7204.307259] ? do_raw_spin_unlock+0x75/0x190 [ 7204.310808] ? _raw_spin_unlock+0x12/0x30 [ 7204.313169] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7204.318907] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7204.322478] ll_atomic_open+0x36a/0xd70 [lustre] [ 7204.328288] lookup_open+0x356/0xb40 [ 7204.329332] do_last+0x313/0xfc0 [ 7204.333762] ? nd_jump_root+0xe5/0x160 [ 7204.338255] ? path_init+0x437/0x520 [ 7204.341409] path_openat+0xf7/0x500 [ 7204.343592] ? do_raw_spin_unlock+0x75/0x190 [ 7204.353612] do_filp_open+0x99/0x140 [ 7204.355004] ? getname_flags+0x6e/0x330 [ 7204.356878] ? __check_object_size+0xff/0x256 [ 7204.358649] ? do_raw_spin_unlock+0x75/0x190 [ 7204.360550] ? _raw_spin_unlock+0x12/0x30 [ 7204.362215] do_sys_openat2+0x2b4/0x410 [ 7204.363760] do_sys_open+0x73/0xa0 [ 7204.365044] __x64_sys_openat+0x24/0x30 [ 7204.366634] do_syscall_64+0xc1/0x440 [ 7204.368268] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7204.371667] RIP: 0033:0x7fdc0afbf332 [ 7204.381573] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7204.393701] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7204.399089] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7204.402081] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7204.410546] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7204.417755] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7204.423399] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7208.048065] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 7208.052106] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7208.071221] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7208.074498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7208.077796] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7208.079606] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7208.085576] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7208.088149] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7208.091510] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7208.093508] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7208.100513] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7208.112331] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7208.116594] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7208.129819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7208.132708] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7208.136131] Call Trace: [ 7208.136886] [ 7208.137522] ? show_regs.cold.9+0x22/0x2f [ 7208.138874] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7208.142711] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7208.144692] ? watchdog+0x40/0x40 [ 7208.146831] ? hrtimer_interrupt+0x128/0x2c0 [ 7208.148218] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7208.150624] ? apic_timer_interrupt+0xf/0x20 [ 7208.152177] [ 7208.153022] ? native_safe_halt+0x2e/0x50 [ 7208.154565] kvm_wait+0xcd/0xe0 [ 7208.156419] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7208.160008] ? dump_stack+0xf6/0x10e [ 7208.161860] do_raw_spin_lock+0x177/0x190 [ 7208.162967] _raw_spin_lock+0x21/0x30 [ 7208.164360] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7208.166832] obd_mod_free+0x46/0x140 [ptlrpc] [ 7208.168769] mdc_commit_open+0x91/0xb0 [mdc] [ 7208.170534] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7208.172780] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7208.175154] after_reply+0xe2c/0x1780 [ptlrpc] [ 7208.176794] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7208.179184] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7208.180747] ? do_wait_intr+0xf0/0xf0 [ 7208.181937] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7208.183672] kthread+0x1d1/0x200 [ 7208.184642] ? set_kthread_struct+0x70/0x70 [ 7208.186143] ret_from_fork+0x1f/0x30 [ 7213.535047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5188s! [ 7213.541633] Showing busy workqueues and worker pools: [ 7213.549860] workqueue events: flags=0x0 [ 7213.552332] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7213.557123] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7213.564084] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7213.572543] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7213.578645] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7213.588773] pending: kfree_rcu_monitor [ 7213.590542] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7213.596482] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7213.610208] workqueue events_long: flags=0x0 [ 7213.614085] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7213.622076] pending: lu_objects_flush [obdclass] [ 7213.627839] workqueue events_unbound: flags=0x2 [ 7213.628988] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7213.631868] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7213.636133] workqueue events_power_efficient: flags=0x80 [ 7213.638360] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7213.641948] pending: neigh_periodic_work, check_lifetime [ 7213.646314] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7213.649372] pending: do_cache_clean, neigh_periodic_work [ 7213.652719] workqueue mm_percpu_wq: flags=0x8 [ 7213.654359] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7213.657034] pending: lru_add_drain_per_cpu [ 7213.659972] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7213.662146] pending: vmstat_update, lru_add_drain_per_cpu [ 7213.666251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7213.669119] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7213.673667] workqueue writeback: flags=0x4a [ 7213.675366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7213.677760] pending: wb_workfn, wb_workfn [ 7213.679393] workqueue xprtiod: flags=0xa [ 7213.681549] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7213.684508] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7213.687221] workqueue ipv6_addrconf: flags=0x40008 [ 7213.688734] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7213.696637] pending: addrconf_verify_work [ 7213.698545] workqueue knbd0-recv: flags=0x1a [ 7213.699933] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7213.702842] in-flight: 655:recv_work [ 7213.707033] workqueue ptlrpc_pinger: flags=0x2 [ 7213.709944] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7213.714121] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7213.717591] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5188s workers=4 idle: 36396 38 11188 [ 7213.722458] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7204s workers=2 idle: 658 [ 7216.051030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7216.053791] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7216.091384] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7216.100065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7216.108221] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7216.112991] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7216.115600] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7216.136948] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7216.146130] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7216.149457] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7216.156361] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7216.162676] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7216.170585] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7216.175922] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7216.180664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7216.182715] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7216.186530] Call Trace: [ 7216.187808] [ 7216.188803] ? show_regs.cold.9+0x22/0x2f [ 7216.190927] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7216.193737] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7216.195950] ? watchdog+0x40/0x40 [ 7216.197720] ? hrtimer_interrupt+0x128/0x2c0 [ 7216.199652] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7216.202088] ? apic_timer_interrupt+0xf/0x20 [ 7216.204836] [ 7216.205810] ? native_safe_halt+0x2e/0x50 [ 7216.207548] kvm_wait+0xcd/0xe0 [ 7216.208896] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7216.217039] do_raw_spin_lock+0x177/0x190 [ 7216.218904] _raw_spin_lock+0x21/0x30 [ 7216.221937] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7216.223751] process_one_work+0x2c8/0x700 [ 7216.228213] worker_thread+0x296/0x6e0 [ 7216.230224] ? rescuer_thread+0x570/0x570 [ 7216.232671] kthread+0x1d1/0x200 [ 7216.234589] ? set_kthread_struct+0x70/0x70 [ 7216.240764] ret_from_fork+0x1f/0x30 [ 7220.054174] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 7220.058110] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7220.084721] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7220.090178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7220.094521] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.097714] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7220.107421] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7220.111264] RAX: 0000000000007f05 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7220.114929] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7220.119373] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7220.123012] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7220.126932] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7220.130719] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7220.135710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7220.138007] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7220.141106] Call Trace: [ 7220.142015] [ 7220.143068] ? show_regs.cold.9+0x22/0x2f [ 7220.145108] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7220.147372] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7220.149383] ? watchdog+0x40/0x40 [ 7220.151230] ? hrtimer_interrupt+0x128/0x2c0 [ 7220.153087] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7220.155503] ? apic_timer_interrupt+0xf/0x20 [ 7220.157672] [ 7220.159024] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.161400] do_raw_spin_lock+0x177/0x190 [ 7220.162921] _raw_spin_lock+0x21/0x30 [ 7220.164952] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7220.168255] ? do_raw_spin_unlock+0x75/0x190 [ 7220.170101] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7220.173159] ? do_wait_intr+0xf0/0xf0 [ 7220.175194] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7220.178121] mdc_close+0x53b/0xd70 [mdc] [ 7220.180613] lmv_close+0x18d/0x490 [lmv] [ 7220.182129] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7220.185253] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7220.188316] ll_md_close+0x274/0x800 [lustre] [ 7220.190916] ll_file_release+0x128/0xc50 [lustre] [ 7220.193236] __fput+0x10d/0x440 [ 7220.194908] ____fput+0x12/0x20 [ 7220.196465] task_work_run+0xc6/0x110 [ 7220.198243] exit_to_usermode_loop+0x1dd/0x1f0 [ 7220.200615] do_syscall_64+0x3d6/0x440 [ 7220.202103] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7220.204050] RIP: 0033:0x7fdc0afbfcd8 [ 7220.206257] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7220.213510] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7220.216088] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7220.218710] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7220.221707] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7220.225200] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7220.228029] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7232.052022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7232.056488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7232.082984] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7232.098266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7232.103164] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7232.105673] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7232.116254] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7232.120573] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7232.123955] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7232.127627] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7232.131424] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7232.135812] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7232.140834] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7232.146815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7232.153420] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7232.157257] Call Trace: [ 7232.159425] [ 7232.161361] ? show_regs.cold.9+0x22/0x2f [ 7232.165064] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7232.168985] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7232.171787] ? watchdog+0x40/0x40 [ 7232.173751] ? hrtimer_interrupt+0x128/0x2c0 [ 7232.176098] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7232.178642] ? apic_timer_interrupt+0xf/0x20 [ 7232.183222] [ 7232.184233] ? native_safe_halt+0x2e/0x50 [ 7232.186295] kvm_wait+0xcd/0xe0 [ 7232.187992] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7232.193585] do_raw_spin_lock+0x177/0x190 [ 7232.195532] _raw_spin_lock+0x21/0x30 [ 7232.203494] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7232.207251] ? do_raw_spin_unlock+0x75/0x190 [ 7232.209609] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7232.212809] ? do_wait_intr+0xf0/0xf0 [ 7232.215092] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7232.222228] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7232.228720] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7232.234181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7232.240743] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7232.244187] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7232.247233] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7232.251187] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7232.258249] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7232.262779] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7232.265853] lmv_intent_open+0x338/0x1150 [lmv] [ 7232.268379] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7232.272145] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7232.276809] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7232.280726] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7232.283648] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7232.287897] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7232.290135] ? do_raw_spin_unlock+0x75/0x190 [ 7232.293110] ? _raw_spin_unlock+0x12/0x30 [ 7232.297258] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7232.302223] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7232.304989] ll_atomic_open+0x36a/0xd70 [lustre] [ 7232.313099] lookup_open+0x356/0xb40 [ 7232.315186] do_last+0x313/0xfc0 [ 7232.316810] ? nd_jump_root+0xe5/0x160 [ 7232.318875] ? path_init+0x437/0x520 [ 7232.322196] path_openat+0xf7/0x500 [ 7232.325535] ? do_raw_spin_unlock+0x75/0x190 [ 7232.327960] do_filp_open+0x99/0x140 [ 7232.329731] ? getname_flags+0x6e/0x330 [ 7232.332969] ? __check_object_size+0xff/0x256 [ 7232.336142] ? do_raw_spin_unlock+0x75/0x190 [ 7232.338221] ? _raw_spin_unlock+0x12/0x30 [ 7232.340730] do_sys_openat2+0x2b4/0x410 [ 7232.347131] do_sys_open+0x73/0xa0 [ 7232.348341] __x64_sys_openat+0x24/0x30 [ 7232.349638] do_syscall_64+0xc1/0x440 [ 7232.350883] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7232.353153] RIP: 0033:0x7fdc0afbf332 [ 7232.355247] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7232.369568] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7232.374638] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7232.379789] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7232.383424] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7232.385854] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7232.391724] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7240.049787] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7240.054997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7240.079521] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7240.083995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7240.089537] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7240.090976] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7240.111116] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7240.123970] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7240.128059] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7240.133930] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7240.140775] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7240.144486] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7240.147856] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7240.151977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7240.156469] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7240.167244] Call Trace: [ 7240.169023] [ 7240.169932] ? show_regs.cold.9+0x22/0x2f [ 7240.172168] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7240.173925] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7240.177122] ? watchdog+0x40/0x40 [ 7240.181166] ? hrtimer_interrupt+0x128/0x2c0 [ 7240.183777] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7240.188766] ? apic_timer_interrupt+0xf/0x20 [ 7240.191628] [ 7240.192232] ? native_safe_halt+0x2e/0x50 [ 7240.196626] kvm_wait+0xcd/0xe0 [ 7240.197972] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7240.208170] ? dump_stack+0xf6/0x10e [ 7240.209104] do_raw_spin_lock+0x177/0x190 [ 7240.210112] _raw_spin_lock+0x21/0x30 [ 7240.222862] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7240.233207] obd_mod_free+0x46/0x140 [ptlrpc] [ 7240.235648] mdc_commit_open+0x91/0xb0 [mdc] [ 7240.239051] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7240.246454] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7240.249915] after_reply+0xe2c/0x1780 [ptlrpc] [ 7240.252853] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7240.257054] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7240.260468] ? do_wait_intr+0xf0/0xf0 [ 7240.261878] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7240.265163] kthread+0x1d1/0x200 [ 7240.266905] ? set_kthread_struct+0x70/0x70 [ 7240.271211] ret_from_fork+0x1f/0x30 [ 7243.744043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5218s! [ 7243.748626] Showing busy workqueues and worker pools: [ 7243.753393] workqueue events: flags=0x0 [ 7243.755422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7243.763723] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7243.772008] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7243.775323] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7243.790426] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7243.797951] pending: kfree_rcu_monitor [ 7243.802395] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7243.808175] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7243.820079] workqueue events_long: flags=0x0 [ 7243.824531] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7243.832910] pending: lu_objects_flush [obdclass] [ 7243.835561] workqueue events_unbound: flags=0x2 [ 7243.842916] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7243.845144] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7243.848510] workqueue events_power_efficient: flags=0x80 [ 7243.850848] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7243.857323] pending: neigh_periodic_work, check_lifetime [ 7243.858990] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7243.860786] pending: do_cache_clean, neigh_periodic_work [ 7243.862469] workqueue mm_percpu_wq: flags=0x8 [ 7243.863566] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7243.870829] pending: lru_add_drain_per_cpu [ 7243.873242] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7243.880332] pending: vmstat_update, lru_add_drain_per_cpu [ 7243.881702] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7243.884992] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7243.888177] workqueue writeback: flags=0x4a [ 7243.889438] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7243.892500] pending: wb_workfn, wb_workfn [ 7243.897716] workqueue xprtiod: flags=0xa [ 7243.898786] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7243.902202] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7243.905578] workqueue ipv6_addrconf: flags=0x40008 [ 7243.907413] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7243.914705] pending: addrconf_verify_work [ 7243.915932] workqueue knbd0-recv: flags=0x1a [ 7243.917042] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7243.925523] in-flight: 655:recv_work [ 7243.929254] workqueue ptlrpc_pinger: flags=0x2 [ 7243.931079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7243.937358] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7243.943343] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5219s workers=4 idle: 36396 38 11188 [ 7243.955536] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7234s workers=2 idle: 658 [ 7244.051048] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7244.055161] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7244.074412] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7244.079184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7244.082927] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7244.085786] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7244.087649] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7244.096668] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7244.099604] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7244.102228] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7244.105312] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7244.107905] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7244.109964] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7244.112269] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7244.115134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7244.117413] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7244.119747] Call Trace: [ 7244.120527] [ 7244.121210] ? show_regs.cold.9+0x22/0x2f [ 7244.122762] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7244.124091] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7244.125345] ? watchdog+0x40/0x40 [ 7244.126588] ? hrtimer_interrupt+0x128/0x2c0 [ 7244.127949] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7244.129782] ? apic_timer_interrupt+0xf/0x20 [ 7244.131341] [ 7244.132116] ? native_safe_halt+0x2e/0x50 [ 7244.133416] kvm_wait+0xcd/0xe0 [ 7244.134601] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7244.136296] do_raw_spin_lock+0x177/0x190 [ 7244.137585] _raw_spin_lock+0x21/0x30 [ 7244.139163] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7244.141389] process_one_work+0x2c8/0x700 [ 7244.143075] worker_thread+0x296/0x6e0 [ 7244.144390] ? rescuer_thread+0x570/0x570 [ 7244.145760] kthread+0x1d1/0x200 [ 7244.146756] ? set_kthread_struct+0x70/0x70 [ 7244.148221] ret_from_fork+0x1f/0x30 [ 7248.054029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 7248.057452] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7248.082056] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7248.087516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7248.091802] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7248.094149] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7248.103764] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7248.108494] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7248.112329] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7248.116469] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7248.120590] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7248.124053] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7248.127994] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7248.132037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7248.135012] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7248.139282] Call Trace: [ 7248.140739] [ 7248.141652] ? show_regs.cold.9+0x22/0x2f [ 7248.143596] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7248.146283] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7248.148327] ? watchdog+0x40/0x40 [ 7248.150221] ? hrtimer_interrupt+0x128/0x2c0 [ 7248.152739] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7248.156969] ? apic_timer_interrupt+0xf/0x20 [ 7248.163154] [ 7248.164927] ? native_safe_halt+0x2e/0x50 [ 7248.168022] kvm_wait+0xcd/0xe0 [ 7248.169852] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7248.177844] do_raw_spin_lock+0x177/0x190 [ 7248.180428] _raw_spin_lock+0x21/0x30 [ 7248.182937] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7248.187708] ? do_raw_spin_unlock+0x75/0x190 [ 7248.191431] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7248.193982] ? do_wait_intr+0xf0/0xf0 [ 7248.195865] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7248.198497] mdc_close+0x53b/0xd70 [mdc] [ 7248.200518] lmv_close+0x18d/0x490 [lmv] [ 7248.202035] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7248.204847] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7248.207279] ll_md_close+0x274/0x800 [lustre] [ 7248.209662] ll_file_release+0x128/0xc50 [lustre] [ 7248.212303] __fput+0x10d/0x440 [ 7248.213773] ____fput+0x12/0x20 [ 7248.214771] task_work_run+0xc6/0x110 [ 7248.216608] exit_to_usermode_loop+0x1dd/0x1f0 [ 7248.218069] do_syscall_64+0x3d6/0x440 [ 7248.219845] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7248.222060] RIP: 0033:0x7fdc0afbfcd8 [ 7248.223886] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7248.232122] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7248.235512] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7248.238926] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7248.242972] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7248.246500] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7248.249908] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7260.058621] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7260.070227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7260.126574] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7260.141990] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7260.153571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7260.160072] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7260.171979] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7260.174761] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7260.178155] RDX: 0000000000003262 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7260.181889] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7260.194718] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7260.202095] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7260.210227] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7260.220926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7260.225019] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7260.229132] Call Trace: [ 7260.230144] [ 7260.230912] ? show_regs.cold.9+0x22/0x2f [ 7260.232344] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7260.244111] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7260.247907] ? watchdog+0x40/0x40 [ 7260.252650] ? hrtimer_interrupt+0x128/0x2c0 [ 7260.263793] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7260.269356] ? apic_timer_interrupt+0xf/0x20 [ 7260.273251] [ 7260.274310] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7260.277470] do_raw_spin_lock+0x177/0x190 [ 7260.281247] _raw_spin_lock+0x21/0x30 [ 7260.285712] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7260.289739] ? do_raw_spin_unlock+0x75/0x190 [ 7260.296532] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7260.300378] ? do_wait_intr+0xf0/0xf0 [ 7260.302327] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7260.306783] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7260.314168] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7260.318183] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7260.322488] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7260.325554] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7260.328237] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7260.330203] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7260.332983] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7260.337305] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7260.340142] lmv_intent_open+0x338/0x1150 [lmv] [ 7260.343818] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7260.350649] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7260.355092] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7260.364628] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7260.372320] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7260.376403] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7260.383829] ? do_raw_spin_unlock+0x75/0x190 [ 7260.389255] ? _raw_spin_unlock+0x12/0x30 [ 7260.392595] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7260.398247] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7260.403721] ll_atomic_open+0x36a/0xd70 [lustre] [ 7260.418675] lookup_open+0x356/0xb40 [ 7260.424291] do_last+0x313/0xfc0 [ 7260.428591] ? nd_jump_root+0xe5/0x160 [ 7260.431021] ? path_init+0x437/0x520 [ 7260.436169] path_openat+0xf7/0x500 [ 7260.438424] ? do_raw_spin_unlock+0x75/0x190 [ 7260.440803] do_filp_open+0x99/0x140 [ 7260.442735] ? getname_flags+0x6e/0x330 [ 7260.445046] ? __check_object_size+0xff/0x256 [ 7260.447465] ? do_raw_spin_unlock+0x75/0x190 [ 7260.449475] ? _raw_spin_unlock+0x12/0x30 [ 7260.451622] do_sys_openat2+0x2b4/0x410 [ 7260.453631] do_sys_open+0x73/0xa0 [ 7260.455359] __x64_sys_openat+0x24/0x30 [ 7260.457225] do_syscall_64+0xc1/0x440 [ 7260.459128] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7260.464237] RIP: 0033:0x7fdc0afbf332 [ 7260.470539] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7260.489691] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7260.498132] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7260.503284] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7260.512394] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7260.519075] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7260.528387] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7268.049911] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7268.070172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7268.144106] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7268.153071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7268.166168] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7268.171122] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7268.181068] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7268.184404] RAX: 0000000000006412 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7268.186476] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7268.189087] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7268.192111] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7268.195642] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7268.198878] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7268.202795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7268.205112] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7268.208751] Call Trace: [ 7268.209685] [ 7268.211039] ? show_regs.cold.9+0x22/0x2f [ 7268.213131] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7268.215481] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7268.217356] ? watchdog+0x40/0x40 [ 7268.218832] ? hrtimer_interrupt+0x128/0x2c0 [ 7268.220580] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7268.222399] ? apic_timer_interrupt+0xf/0x20 [ 7268.224437] [ 7268.225195] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7268.227400] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7268.229986] ? dump_stack+0xf6/0x10e [ 7268.231604] do_raw_spin_lock+0x177/0x190 [ 7268.233553] _raw_spin_lock+0x21/0x30 [ 7268.235040] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7268.237494] obd_mod_free+0x46/0x140 [ptlrpc] [ 7268.239671] mdc_commit_open+0x91/0xb0 [mdc] [ 7268.241519] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7268.243637] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7268.246093] after_reply+0xe2c/0x1780 [ptlrpc] [ 7268.248162] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7268.250408] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7268.252388] ? do_wait_intr+0xf0/0xf0 [ 7268.253951] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7268.256496] kthread+0x1d1/0x200 [ 7268.257875] ? set_kthread_struct+0x70/0x70 [ 7268.259281] ret_from_fork+0x1f/0x30 [ 7272.051029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7272.060104] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7272.126744] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7272.136897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7272.149137] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7272.154510] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7272.157364] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7272.176105] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7272.184305] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7272.188400] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7272.193929] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7272.199369] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7272.204739] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7272.208705] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7272.216621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7272.220884] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7272.224304] Call Trace: [ 7272.225638] [ 7272.226900] ? show_regs.cold.9+0x22/0x2f [ 7272.228883] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7272.231255] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7272.235048] ? watchdog+0x40/0x40 [ 7272.238477] ? hrtimer_interrupt+0x128/0x2c0 [ 7272.243002] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7272.246763] ? apic_timer_interrupt+0xf/0x20 [ 7272.250756] [ 7272.251909] ? native_safe_halt+0x2e/0x50 [ 7272.254932] kvm_wait+0xcd/0xe0 [ 7272.257136] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7272.260887] do_raw_spin_lock+0x177/0x190 [ 7272.264806] _raw_spin_lock+0x21/0x30 [ 7272.266679] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7272.271907] process_one_work+0x2c8/0x700 [ 7272.274843] worker_thread+0x296/0x6e0 [ 7272.276470] ? rescuer_thread+0x570/0x570 [ 7272.282266] kthread+0x1d1/0x200 [ 7272.284530] ? set_kthread_struct+0x70/0x70 [ 7272.287155] ret_from_fork+0x1f/0x30 [ 7274.469253] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5249s! [ 7274.482923] Showing busy workqueues and worker pools: [ 7274.484954] workqueue events: flags=0x0 [ 7274.491470] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7274.500491] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7274.509917] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7274.517856] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7274.528009] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7274.530307] pending: kfree_rcu_monitor [ 7274.532082] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7274.535082] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7274.540392] workqueue events_long: flags=0x0 [ 7274.542043] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7274.545216] pending: lu_objects_flush [obdclass] [ 7274.547524] workqueue events_unbound: flags=0x2 [ 7274.549098] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7274.551260] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7274.555569] workqueue events_power_efficient: flags=0x80 [ 7274.557649] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7274.560345] pending: neigh_periodic_work, check_lifetime [ 7274.562844] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7274.565589] pending: do_cache_clean, neigh_periodic_work [ 7274.568014] workqueue mm_percpu_wq: flags=0x8 [ 7274.570253] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7274.572975] pending: lru_add_drain_per_cpu [ 7274.574670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7274.577304] pending: vmstat_update, lru_add_drain_per_cpu [ 7274.579832] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7274.582611] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7274.585073] workqueue writeback: flags=0x4a [ 7274.586624] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7274.588983] pending: wb_workfn, wb_workfn [ 7274.590303] workqueue xprtiod: flags=0xa [ 7274.591512] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7274.593699] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7274.597245] workqueue ipv6_addrconf: flags=0x40008 [ 7274.599074] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7274.606333] pending: addrconf_verify_work [ 7274.612917] workqueue knbd0-recv: flags=0x1a [ 7274.614454] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7274.619660] in-flight: 655:recv_work [ 7274.621206] workqueue ptlrpc_pinger: flags=0x2 [ 7274.626012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7274.630663] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7274.635454] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5249s workers=4 idle: 36396 38 11188 [ 7274.640582] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7265s workers=2 idle: 658 [ 7276.055756] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7276.064082] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7276.115546] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7276.126363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7276.147441] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7276.153682] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7276.184710] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7276.195212] RAX: 0000000000006e08 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7276.203278] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7276.207582] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7276.214021] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7276.220979] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7276.224829] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7276.228579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7276.231522] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7276.235912] Call Trace: [ 7276.238346] [ 7276.239382] ? show_regs.cold.9+0x22/0x2f [ 7276.241344] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7276.243993] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7276.254162] ? watchdog+0x40/0x40 [ 7276.257067] ? hrtimer_interrupt+0x128/0x2c0 [ 7276.261416] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7276.264360] ? apic_timer_interrupt+0xf/0x20 [ 7276.270406] [ 7276.271939] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7276.275574] do_raw_spin_lock+0x177/0x190 [ 7276.277041] _raw_spin_lock+0x21/0x30 [ 7276.278587] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7276.281155] ? do_raw_spin_unlock+0x75/0x190 [ 7276.282777] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7276.284847] ? do_wait_intr+0xf0/0xf0 [ 7276.286367] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7276.289180] mdc_close+0x53b/0xd70 [mdc] [ 7276.291282] lmv_close+0x18d/0x490 [lmv] [ 7276.293823] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7276.296484] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7276.299795] ll_md_close+0x274/0x800 [lustre] [ 7276.301858] ll_file_release+0x128/0xc50 [lustre] [ 7276.304256] __fput+0x10d/0x440 [ 7276.306247] ____fput+0x12/0x20 [ 7276.307406] task_work_run+0xc6/0x110 [ 7276.308655] exit_to_usermode_loop+0x1dd/0x1f0 [ 7276.310528] do_syscall_64+0x3d6/0x440 [ 7276.312174] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7276.314545] RIP: 0033:0x7fdc0afbfcd8 [ 7276.316297] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7276.324947] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7276.327943] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7276.331283] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7276.335221] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7276.341188] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7276.350275] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7288.052823] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7288.056577] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7288.090427] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7288.103242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7288.111874] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7288.118279] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7288.133583] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7288.138013] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7288.141174] RDX: 0000000000005e44 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7288.144037] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7288.151214] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7288.157958] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7288.162118] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7288.165240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7288.168331] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7288.172469] Call Trace: [ 7288.175773] [ 7288.176928] ? show_regs.cold.9+0x22/0x2f [ 7288.181105] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7288.183306] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7288.185271] ? watchdog+0x40/0x40 [ 7288.187240] ? hrtimer_interrupt+0x128/0x2c0 [ 7288.189396] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7288.191769] ? apic_timer_interrupt+0xf/0x20 [ 7288.197571] [ 7288.198556] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7288.204983] do_raw_spin_lock+0x177/0x190 [ 7288.210322] _raw_spin_lock+0x21/0x30 [ 7288.212563] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7288.216448] ? do_raw_spin_unlock+0x75/0x190 [ 7288.218698] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7288.222439] ? do_wait_intr+0xf0/0xf0 [ 7288.223439] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7288.226128] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7288.229034] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7288.232566] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7288.235038] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7288.240538] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7288.243308] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7288.246356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7288.250105] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7288.252406] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7288.257467] lmv_intent_open+0x338/0x1150 [lmv] [ 7288.260113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7288.264452] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7288.270838] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7288.274549] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7288.276192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7288.286484] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7288.291383] ? do_raw_spin_unlock+0x75/0x190 [ 7288.299376] ? _raw_spin_unlock+0x12/0x30 [ 7288.306150] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7288.311585] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7288.316371] ll_atomic_open+0x36a/0xd70 [lustre] [ 7288.323158] lookup_open+0x356/0xb40 [ 7288.326795] do_last+0x313/0xfc0 [ 7288.328698] ? nd_jump_root+0xe5/0x160 [ 7288.331767] ? path_init+0x437/0x520 [ 7288.335270] path_openat+0xf7/0x500 [ 7288.338960] ? do_raw_spin_unlock+0x75/0x190 [ 7288.341534] do_filp_open+0x99/0x140 [ 7288.344165] ? getname_flags+0x6e/0x330 [ 7288.346967] ? __check_object_size+0xff/0x256 [ 7288.350115] ? do_raw_spin_unlock+0x75/0x190 [ 7288.352350] ? _raw_spin_unlock+0x12/0x30 [ 7288.357214] do_sys_openat2+0x2b4/0x410 [ 7288.360786] do_sys_open+0x73/0xa0 [ 7288.368139] __x64_sys_openat+0x24/0x30 [ 7288.371918] do_syscall_64+0xc1/0x440 [ 7288.373997] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7288.377774] RIP: 0033:0x7fdc0afbf332 [ 7288.381833] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7288.408688] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7288.412221] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7288.417170] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7288.422575] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7288.426717] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7288.432076] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7299.458082] rcu: INFO: rcu_sched self-detected stall on CPU [ 7299.465524] rcu: 0-...!: (3243005 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 7299.476269] (t=5280115 jiffies g=270761 q=3923) [ 7299.483333] rcu: rcu_sched kthread starved for 5275047 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 7299.494454] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7299.508396] rcu: RCU grace-period kthread stack dump: [ 7299.513245] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7299.521955] Call Trace: [ 7299.523243] __schedule+0x351/0xcb0 [ 7299.526208] schedule+0xc0/0x180 [ 7299.527519] schedule_timeout+0xb4/0x190 [ 7299.533988] ? do_raw_spin_unlock+0x75/0x190 [ 7299.539341] ? __next_timer_interrupt+0x160/0x160 [ 7299.548544] ? prepare_to_swait_event+0xf3/0x140 [ 7299.550292] rcu_gp_kthread+0x8dd/0x15b0 [ 7299.557376] ? force_qs_rnp+0x2c0/0x2c0 [ 7299.566640] kthread+0x1d1/0x200 [ 7299.578304] ? set_kthread_struct+0x70/0x70 [ 7299.579809] ret_from_fork+0x1f/0x30 [ 7299.580973] rcu: Stack dump where RCU GP kthread last ran: [ 7299.598469] Sending NMI from CPU 0 to CPUs 1: [ 7299.600705] NMI backtrace for cpu 1 [ 7299.600761] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7299.600764] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7299.600766] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7299.600827] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7299.600832] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7299.600834] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 [ 7299.600836] RAX: 0000000000000ebc RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7299.600838] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7299.600840] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7299.600841] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7299.600842] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7299.600844] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7299.600846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7299.600847] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7299.600848] Call Trace: [ 7299.600849] [ 7299.600850] ? show_regs.cold.9+0x22/0x2f [ 7299.600851] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7299.600852] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7299.600853] ? nmi_handle+0xbd/0x210 [ 7299.600865] ? default_do_nmi+0x4b/0x1b0 [ 7299.600866] ? do_nmi+0x280/0x380 [ 7299.600867] ? end_repeat_nmi+0x16/0x1f [ 7299.600868] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7299.600870] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7299.600871] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7299.600872] [ 7299.600873] do_raw_spin_lock+0x177/0x190 [ 7299.600874] _raw_spin_lock+0x21/0x30 [ 7299.600875] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7299.600876] process_one_work+0x2c8/0x700 [ 7299.600877] worker_thread+0x296/0x6e0 [ 7299.600878] ? rescuer_thread+0x570/0x570 [ 7299.600879] kthread+0x1d1/0x200 [ 7299.600880] ? set_kthread_struct+0x70/0x70 [ 7299.600881] ret_from_fork+0x1f/0x30 [ 7299.604124] NMI backtrace for cpu 0 [ 7299.749734] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7299.771135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7299.787427] Call Trace: [ 7299.791549] [ 7299.794623] ? dump_stack+0xbb/0x10e [ 7299.799167] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7299.801213] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7299.803159] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7299.805151] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7299.807282] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7299.809558] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7299.813235] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7299.815513] ? update_process_times+0x6d/0xc0 [ 7299.817857] ? tick_sched_handle.isra.18+0x27/0x90 [ 7299.819987] ? tick_sched_timer+0x53/0xb0 [ 7299.821501] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7299.826858] ? tick_sched_handle.isra.18+0x90/0x90 [ 7299.832455] ? hrtimer_interrupt+0x128/0x2c0 [ 7299.836068] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7299.840160] ? apic_timer_interrupt+0xf/0x20 [ 7299.842428] [ 7299.843995] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7299.846625] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7299.850742] ? dump_stack+0xf6/0x10e [ 7299.853220] ? do_raw_spin_lock+0x177/0x190 [ 7299.856233] ? _raw_spin_lock+0x21/0x30 [ 7299.860876] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7299.865129] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7299.869789] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7299.873557] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7299.876176] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7299.879118] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7299.884603] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7299.887435] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7299.890180] ? do_wait_intr+0xf0/0xf0 [ 7299.891748] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7299.893557] ? kthread+0x1d1/0x200 [ 7299.897722] ? set_kthread_struct+0x70/0x70 [ 7299.899794] ? ret_from_fork+0x1f/0x30 [ 7304.054014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7304.056862] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7304.075400] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7304.081551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7304.085532] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7304.088325] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7304.096399] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7304.099778] RAX: 0000000000003e0c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7304.102817] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7304.105355] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7304.108063] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7304.111212] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7304.113953] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7304.117324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7304.119811] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7304.122828] Call Trace: [ 7304.124040] [ 7304.124992] ? show_regs.cold.9+0x22/0x2f [ 7304.126866] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7304.128934] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7304.131260] ? watchdog+0x40/0x40 [ 7304.132515] ? hrtimer_interrupt+0x128/0x2c0 [ 7304.134215] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7304.135752] ? apic_timer_interrupt+0xf/0x20 [ 7304.137735] [ 7304.138554] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7304.140924] do_raw_spin_lock+0x177/0x190 [ 7304.142884] _raw_spin_lock+0x21/0x30 [ 7304.144915] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7304.147363] ? do_raw_spin_unlock+0x75/0x190 [ 7304.149398] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7304.152529] ? do_wait_intr+0xf0/0xf0 [ 7304.156076] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7304.162317] mdc_close+0x53b/0xd70 [mdc] [ 7304.164224] lmv_close+0x18d/0x490 [lmv] [ 7304.165902] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7304.168582] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7304.170956] ll_md_close+0x274/0x800 [lustre] [ 7304.173081] ll_file_release+0x128/0xc50 [lustre] [ 7304.175401] __fput+0x10d/0x440 [ 7304.176733] ____fput+0x12/0x20 [ 7304.178317] task_work_run+0xc6/0x110 [ 7304.180017] exit_to_usermode_loop+0x1dd/0x1f0 [ 7304.182337] do_syscall_64+0x3d6/0x440 [ 7304.184079] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7304.186848] RIP: 0033:0x7fdc0afbfcd8 [ 7304.188722] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7304.198652] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7304.203117] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7304.207514] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7304.215840] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7304.221815] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7304.230216] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7304.671628] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5279s! [ 7304.682235] Showing busy workqueues and worker pools: [ 7304.685820] workqueue events: flags=0x0 [ 7304.693477] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7304.695138] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7304.705495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7304.709720] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7304.716093] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7304.718008] pending: kfree_rcu_monitor [ 7304.719116] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7304.720915] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7304.727537] workqueue events_long: flags=0x0 [ 7304.728947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7304.731014] pending: lu_objects_flush [obdclass] [ 7304.735046] workqueue events_unbound: flags=0x2 [ 7304.736254] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7304.741797] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7304.746344] workqueue events_power_efficient: flags=0x80 [ 7304.750673] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7304.754477] pending: neigh_periodic_work, check_lifetime [ 7304.755924] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7304.770383] pending: do_cache_clean, neigh_periodic_work [ 7304.772342] workqueue mm_percpu_wq: flags=0x8 [ 7304.777992] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7304.780019] pending: lru_add_drain_per_cpu [ 7304.783481] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7304.796456] pending: vmstat_update, lru_add_drain_per_cpu [ 7304.801041] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7304.805341] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7304.809373] workqueue writeback: flags=0x4a [ 7304.812296] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7304.817139] pending: wb_workfn, wb_workfn [ 7304.819814] workqueue xprtiod: flags=0xa [ 7304.824640] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7304.831479] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7304.836628] workqueue ipv6_addrconf: flags=0x40008 [ 7304.838348] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7304.843559] pending: addrconf_verify_work [ 7304.846008] workqueue knbd0-recv: flags=0x1a [ 7304.847320] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7304.850169] in-flight: 655:recv_work [ 7304.851627] workqueue ptlrpc_pinger: flags=0x2 [ 7304.854140] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7304.856786] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7304.859529] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5279s workers=4 idle: 36396 38 11188 [ 7304.862272] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7295s workers=2 idle: 658 [ 7316.052090] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7316.056288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7316.095498] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7316.102224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7316.113847] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7316.119077] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7316.132803] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7316.139332] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7316.147960] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7316.153220] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7316.156444] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7316.159443] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7316.162914] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7316.172946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7316.175731] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7316.178267] Call Trace: [ 7316.178956] [ 7316.182527] ? show_regs.cold.9+0x22/0x2f [ 7316.187175] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7316.189188] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7316.192354] ? watchdog+0x40/0x40 [ 7316.194324] ? hrtimer_interrupt+0x128/0x2c0 [ 7316.196707] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7316.202207] ? apic_timer_interrupt+0xf/0x20 [ 7316.205424] [ 7316.206845] ? native_safe_halt+0x2e/0x50 [ 7316.208841] kvm_wait+0xcd/0xe0 [ 7316.211182] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7316.215997] do_raw_spin_lock+0x177/0x190 [ 7316.219231] _raw_spin_lock+0x21/0x30 [ 7316.221347] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7316.227970] ? do_raw_spin_unlock+0x75/0x190 [ 7316.231969] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7316.235713] ? do_wait_intr+0xf0/0xf0 [ 7316.237459] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7316.244990] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7316.247012] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7316.259441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7316.263895] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7316.273182] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7316.277019] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7316.279555] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7316.282906] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7316.285956] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7316.289643] lmv_intent_open+0x338/0x1150 [lmv] [ 7316.297064] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7316.301266] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7316.307890] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7316.311995] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7316.317466] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7316.322665] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7316.324897] ? do_raw_spin_unlock+0x75/0x190 [ 7316.328051] ? _raw_spin_unlock+0x12/0x30 [ 7316.333104] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7316.337448] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7316.346556] ll_atomic_open+0x36a/0xd70 [lustre] [ 7316.353925] lookup_open+0x356/0xb40 [ 7316.355225] do_last+0x313/0xfc0 [ 7316.356350] ? nd_jump_root+0xe5/0x160 [ 7316.364570] ? path_init+0x437/0x520 [ 7316.366802] path_openat+0xf7/0x500 [ 7316.368464] ? do_raw_spin_unlock+0x75/0x190 [ 7316.370026] do_filp_open+0x99/0x140 [ 7316.372227] ? getname_flags+0x6e/0x330 [ 7316.375133] ? __check_object_size+0xff/0x256 [ 7316.377693] ? do_raw_spin_unlock+0x75/0x190 [ 7316.388802] ? _raw_spin_unlock+0x12/0x30 [ 7316.394932] do_sys_openat2+0x2b4/0x410 [ 7316.397221] do_sys_open+0x73/0xa0 [ 7316.399996] __x64_sys_openat+0x24/0x30 [ 7316.403973] do_syscall_64+0xc1/0x440 [ 7316.407910] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7316.411140] RIP: 0033:0x7fdc0afbf332 [ 7316.415416] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7316.429539] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7316.434812] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7316.439718] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7316.444977] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7316.450609] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7316.453876] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7324.056100] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7324.067217] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7324.102383] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7324.108135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7324.112428] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7324.119891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7324.123129] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7324.140863] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7324.145935] RAX: 0000000000006c4b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7324.152133] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7324.159143] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7324.163208] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7324.169575] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7324.177541] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7324.188117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7324.192454] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7324.200529] Call Trace: [ 7324.202122] [ 7324.203026] ? show_regs.cold.9+0x22/0x2f [ 7324.208000] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7324.211066] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7324.219560] ? watchdog+0x40/0x40 [ 7324.228408] ? hrtimer_interrupt+0x128/0x2c0 [ 7324.233998] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7324.239578] ? apic_timer_interrupt+0xf/0x20 [ 7324.241240] [ 7324.246403] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7324.251015] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7324.254171] do_raw_spin_lock+0x177/0x190 [ 7324.256898] _raw_spin_lock+0x21/0x30 [ 7324.257860] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7324.259375] process_one_work+0x2c8/0x700 [ 7324.263734] worker_thread+0x296/0x6e0 [ 7324.265952] ? rescuer_thread+0x570/0x570 [ 7324.268408] kthread+0x1d1/0x200 [ 7324.269933] ? set_kthread_struct+0x70/0x70 [ 7324.271905] ret_from_fork+0x1f/0x30 [ 7332.049359] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7332.052893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7332.054575] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7332.078018] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7332.078032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7332.078036] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.078056] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7332.078060] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7332.078066] RAX: 0000000000007f85 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7332.078068] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7332.078070] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7332.078072] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7332.078074] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7332.078077] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7332.078079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7332.085934] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7332.086017] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7332.086025] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7332.086027] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.086042] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7332.086047] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7332.086052] RAX: 0000000000007dc2 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7332.086054] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7332.086056] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7332.086058] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7332.086061] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7332.086064] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7332.086067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7332.086069] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7332.086127] Call Trace: [ 7332.086134] [ 7332.086140] ? show_regs.cold.9+0x22/0x2f [ 7332.086150] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7332.086157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7332.086162] ? watchdog+0x40/0x40 [ 7332.086166] ? hrtimer_interrupt+0x128/0x2c0 [ 7332.086170] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7332.086178] ? apic_timer_interrupt+0xf/0x20 [ 7332.086183] [ 7332.086186] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.086193] do_raw_spin_lock+0x177/0x190 [ 7332.086198] _raw_spin_lock+0x21/0x30 [ 7332.086203] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7332.086484] ? do_raw_spin_unlock+0x75/0x190 [ 7332.086505] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7332.086711] ? do_wait_intr+0xf0/0xf0 [ 7332.086719] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7332.086926] mdc_close+0x53b/0xd70 [mdc] [ 7332.086964] lmv_close+0x18d/0x490 [lmv] [ 7332.086996] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7332.088001] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7332.091880] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7332.091895] Call Trace: [ 7332.091903] [ 7332.091908] ? show_regs.cold.9+0x22/0x2f [ 7332.091920] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7332.091927] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7332.091932] ? watchdog+0x40/0x40 [ 7332.091936] ? hrtimer_interrupt+0x128/0x2c0 [ 7332.091940] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7332.091951] ? apic_timer_interrupt+0xf/0x20 [ 7332.091956] [ 7332.091959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.091966] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7332.091970] ? dump_stack+0xf6/0x10e [ 7332.091977] do_raw_spin_lock+0x177/0x190 [ 7332.091982] _raw_spin_lock+0x21/0x30 [ 7332.091990] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7332.092674] obd_mod_free+0x46/0x140 [ptlrpc] [ 7332.092885] mdc_commit_open+0x91/0xb0 [mdc] [ 7332.092927] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7332.093206] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7332.093466] after_reply+0xe2c/0x1780 [ptlrpc] [ 7332.093667] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7332.093859] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7332.098706] ll_md_close+0x274/0x800 [lustre] [ 7332.098834] ll_file_release+0x128/0xc50 [lustre] [ 7332.098917] __fput+0x10d/0x440 [ 7332.098941] ____fput+0x12/0x20 [ 7332.098946] task_work_run+0xc6/0x110 [ 7332.098953] exit_to_usermode_loop+0x1dd/0x1f0 [ 7332.098962] do_syscall_64+0x3d6/0x440 [ 7332.098967] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7332.098976] RIP: 0033:0x7fdc0afbfcd8 [ 7332.098983] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7332.098987] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7332.098992] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7332.098994] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7332.098996] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7332.098999] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7332.099001] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7332.455267] ? do_wait_intr+0xf0/0xf0 [ 7332.458720] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7332.461767] kthread+0x1d1/0x200 [ 7332.463317] ? set_kthread_struct+0x70/0x70 [ 7332.465306] ret_from_fork+0x1f/0x30 [ 7334.879258] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5309s! [ 7334.894616] Showing busy workqueues and worker pools: [ 7334.896230] workqueue events: flags=0x0 [ 7334.899289] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7334.905666] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7334.917619] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7334.922211] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7334.927333] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7334.932208] pending: kfree_rcu_monitor [ 7334.936625] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7334.938678] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7334.945869] workqueue events_long: flags=0x0 [ 7334.947122] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7334.949619] pending: lu_objects_flush [obdclass] [ 7334.960717] workqueue events_unbound: flags=0x2 [ 7334.966601] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7334.971619] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7334.980914] workqueue events_power_efficient: flags=0x80 [ 7334.984572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7334.990864] pending: neigh_periodic_work, check_lifetime [ 7334.993730] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7334.996537] pending: do_cache_clean, neigh_periodic_work [ 7334.998932] workqueue mm_percpu_wq: flags=0x8 [ 7335.000016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7335.001617] pending: lru_add_drain_per_cpu [ 7335.008713] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7335.011990] pending: vmstat_update, lru_add_drain_per_cpu [ 7335.013962] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7335.015893] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7335.027112] workqueue writeback: flags=0x4a [ 7335.029188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7335.035720] pending: wb_workfn, wb_workfn [ 7335.042607] workqueue xprtiod: flags=0xa [ 7335.046537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7335.052736] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7335.062343] workqueue ipv6_addrconf: flags=0x40008 [ 7335.070244] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7335.077543] pending: addrconf_verify_work [ 7335.081097] workqueue knbd0-recv: flags=0x1a [ 7335.083356] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7335.090899] in-flight: 655:recv_work [ 7335.095746] workqueue ptlrpc_pinger: flags=0x2 [ 7335.097472] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7335.100244] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7335.104106] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5310s workers=4 idle: 36396 38 11188 [ 7335.108489] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7325s workers=2 idle: 658 [ 7344.056736] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7344.059299] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7344.106093] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7344.110981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7344.114033] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7344.115932] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7344.132823] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7344.136342] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7344.139699] RDX: 0000000000003318 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7344.142739] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7344.145673] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7344.148240] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7344.151013] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7344.155687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7344.158126] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7344.161453] Call Trace: [ 7344.162477] [ 7344.163318] ? show_regs.cold.9+0x22/0x2f [ 7344.165083] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7344.167557] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7344.169524] ? watchdog+0x40/0x40 [ 7344.170983] ? hrtimer_interrupt+0x128/0x2c0 [ 7344.172606] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7344.174367] ? apic_timer_interrupt+0xf/0x20 [ 7344.175996] [ 7344.176807] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7344.179058] do_raw_spin_lock+0x177/0x190 [ 7344.180601] _raw_spin_lock+0x21/0x30 [ 7344.182097] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7344.184223] ? do_raw_spin_unlock+0x75/0x190 [ 7344.185789] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7344.187761] ? do_wait_intr+0xf0/0xf0 [ 7344.189252] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7344.191238] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7344.193540] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7344.195619] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7344.197432] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7344.199362] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7344.200954] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7344.202378] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7344.204119] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7344.206458] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7344.208297] lmv_intent_open+0x338/0x1150 [lmv] [ 7344.209740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7344.211673] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7344.213675] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7344.215335] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7344.217406] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7344.221767] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7344.225469] ? do_raw_spin_unlock+0x75/0x190 [ 7344.227441] ? _raw_spin_unlock+0x12/0x30 [ 7344.229661] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7344.232559] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7344.234974] ll_atomic_open+0x36a/0xd70 [lustre] [ 7344.237336] lookup_open+0x356/0xb40 [ 7344.239262] do_last+0x313/0xfc0 [ 7344.240935] ? nd_jump_root+0xe5/0x160 [ 7344.242756] ? path_init+0x437/0x520 [ 7344.244099] path_openat+0xf7/0x500 [ 7344.245822] ? do_raw_spin_unlock+0x75/0x190 [ 7344.247752] do_filp_open+0x99/0x140 [ 7344.249489] ? getname_flags+0x6e/0x330 [ 7344.251087] ? __check_object_size+0xff/0x256 [ 7344.253513] ? do_raw_spin_unlock+0x75/0x190 [ 7344.255996] ? _raw_spin_unlock+0x12/0x30 [ 7344.257970] do_sys_openat2+0x2b4/0x410 [ 7344.260131] do_sys_open+0x73/0xa0 [ 7344.261668] __x64_sys_openat+0x24/0x30 [ 7344.263327] do_syscall_64+0xc1/0x440 [ 7344.265038] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7344.267770] RIP: 0033:0x7fdc0afbf332 [ 7344.269469] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7344.278158] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7344.281971] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7344.284340] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7344.287173] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7344.293644] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7344.300369] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7352.051018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7352.053158] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7352.067576] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7352.081856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7352.084660] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7352.087707] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7352.089707] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7352.096146] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7352.099336] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7352.101926] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7352.104151] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7352.107036] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7352.110018] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7352.113349] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7352.117055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7352.119034] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7352.122048] Call Trace: [ 7352.123206] [ 7352.124047] ? show_regs.cold.9+0x22/0x2f [ 7352.125829] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7352.127813] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7352.130000] ? watchdog+0x40/0x40 [ 7352.131018] ? hrtimer_interrupt+0x128/0x2c0 [ 7352.132504] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7352.134251] ? apic_timer_interrupt+0xf/0x20 [ 7352.135914] [ 7352.136595] ? native_safe_halt+0x2e/0x50 [ 7352.137711] kvm_wait+0xcd/0xe0 [ 7352.138587] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7352.140099] do_raw_spin_lock+0x177/0x190 [ 7352.141538] _raw_spin_lock+0x21/0x30 [ 7352.142786] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7352.144417] process_one_work+0x2c8/0x700 [ 7352.145506] worker_thread+0x296/0x6e0 [ 7352.146554] ? rescuer_thread+0x570/0x570 [ 7352.147575] kthread+0x1d1/0x200 [ 7352.148476] ? set_kthread_struct+0x70/0x70 [ 7352.149954] ret_from_fork+0x1f/0x30 [ 7360.054113] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7360.055307] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7360.060017] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7360.060095] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7360.060103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7360.060105] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7360.060119] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7360.060123] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7360.060127] RAX: 0000000000006e23 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7360.060130] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7360.060133] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7360.060135] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7360.060137] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7360.060141] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7360.060143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7360.060146] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7360.060155] Call Trace: [ 7360.060160] [ 7360.060167] ? show_regs.cold.9+0x22/0x2f [ 7360.060176] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7360.060183] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7360.061124] ? watchdog+0x40/0x40 [ 7360.061133] ? hrtimer_interrupt+0x128/0x2c0 [ 7360.061138] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7360.061147] ? apic_timer_interrupt+0xf/0x20 [ 7360.061152] [ 7360.061156] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7360.061165] do_raw_spin_lock+0x177/0x190 [ 7360.061170] _raw_spin_lock+0x21/0x30 [ 7360.061175] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7360.061920] ? do_raw_spin_unlock+0x75/0x190 [ 7360.061930] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7360.062142] ? do_wait_intr+0xf0/0xf0 [ 7360.062151] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7360.062578] mdc_close+0x53b/0xd70 [mdc] [ 7360.063019] lmv_close+0x18d/0x490 [lmv] [ 7360.063053] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7360.063162] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7360.064652] ll_md_close+0x274/0x800 [lustre] [ 7360.064754] ll_file_release+0x128/0xc50 [lustre] [ 7360.083106] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7360.083332] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7360.083339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7360.083342] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7360.083356] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7360.083360] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7360.083365] RAX: 0000000000007d1c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7360.083368] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7360.083370] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7360.083372] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7360.083374] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7360.083377] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7360.083443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7360.083445] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7360.083454] Call Trace: [ 7360.083459] [ 7360.083464] ? show_regs.cold.9+0x22/0x2f [ 7360.083474] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7360.083489] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7360.083493] ? watchdog+0x40/0x40 [ 7360.083498] ? hrtimer_interrupt+0x128/0x2c0 [ 7360.083501] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7360.083509] ? apic_timer_interrupt+0xf/0x20 [ 7360.083514] [ 7360.083517] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7360.083522] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7360.083525] ? dump_stack+0xf6/0x10e [ 7360.083531] do_raw_spin_lock+0x177/0x190 [ 7360.083536] _raw_spin_lock+0x21/0x30 [ 7360.083542] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7360.084102] obd_mod_free+0x46/0x140 [ptlrpc] [ 7360.084472] mdc_commit_open+0x91/0xb0 [mdc] [ 7360.084517] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7360.145568] __fput+0x10d/0x440 [ 7360.145602] ____fput+0x12/0x20 [ 7360.145608] task_work_run+0xc6/0x110 [ 7360.145622] exit_to_usermode_loop+0x1dd/0x1f0 [ 7360.145630] do_syscall_64+0x3d6/0x440 [ 7360.145635] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7360.145644] RIP: 0033:0x7fdc0afbfcd8 [ 7360.145653] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7360.145657] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7360.145663] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7360.145665] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7360.145668] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7360.145670] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7360.145672] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7360.557106] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7360.562248] after_reply+0xe2c/0x1780 [ptlrpc] [ 7360.566358] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7360.570075] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7360.574235] ? do_wait_intr+0xf0/0xf0 [ 7360.576033] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7360.578773] kthread+0x1d1/0x200 [ 7360.582884] ? set_kthread_struct+0x70/0x70 [ 7360.585003] ret_from_fork+0x1f/0x30 [ 7365.599431] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5340s! [ 7365.608440] Showing busy workqueues and worker pools: [ 7365.611773] workqueue events: flags=0x0 [ 7365.614732] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7365.620797] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7365.629748] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7365.633929] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7365.641732] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7365.648344] pending: kfree_rcu_monitor [ 7365.651108] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7365.653765] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7365.661608] workqueue events_long: flags=0x0 [ 7365.667175] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7365.675582] pending: lu_objects_flush [obdclass] [ 7365.678294] workqueue events_unbound: flags=0x2 [ 7365.685997] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7365.689626] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7365.696836] workqueue events_power_efficient: flags=0x80 [ 7365.702729] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7365.707618] pending: neigh_periodic_work, check_lifetime [ 7365.710771] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7365.715973] pending: do_cache_clean, neigh_periodic_work [ 7365.719285] workqueue mm_percpu_wq: flags=0x8 [ 7365.727696] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7365.737131] pending: lru_add_drain_per_cpu [ 7365.740391] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7365.744965] pending: vmstat_update, lru_add_drain_per_cpu [ 7365.750456] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7365.752214] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7365.758624] workqueue writeback: flags=0x4a [ 7365.759852] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7365.762450] pending: wb_workfn, wb_workfn [ 7365.763908] workqueue xprtiod: flags=0xa [ 7365.767580] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7365.770614] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7365.775571] workqueue ipv6_addrconf: flags=0x40008 [ 7365.779944] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7365.790710] pending: addrconf_verify_work [ 7365.797587] workqueue knbd0-recv: flags=0x1a [ 7365.799787] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7365.813742] in-flight: 655:recv_work [ 7365.818966] workqueue ptlrpc_pinger: flags=0x2 [ 7365.827557] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7365.831436] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7365.833562] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5340s workers=4 idle: 36396 38 11188 [ 7365.846195] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7356s workers=2 idle: 658 [ 7372.054253] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7372.061633] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7372.110956] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7372.122331] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7372.127374] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7372.129744] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7372.140236] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7372.144825] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7372.153843] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7372.159511] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7372.165045] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7372.169683] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7372.173417] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7372.176397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7372.178678] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7372.183031] Call Trace: [ 7372.184333] [ 7372.185187] ? show_regs.cold.9+0x22/0x2f [ 7372.187881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7372.190320] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7372.193559] ? watchdog+0x40/0x40 [ 7372.194624] ? hrtimer_interrupt+0x128/0x2c0 [ 7372.197708] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7372.200823] ? apic_timer_interrupt+0xf/0x20 [ 7372.203833] [ 7372.205702] ? native_safe_halt+0x2e/0x50 [ 7372.210289] kvm_wait+0xcd/0xe0 [ 7372.212156] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7372.219862] do_raw_spin_lock+0x177/0x190 [ 7372.223023] _raw_spin_lock+0x21/0x30 [ 7372.228742] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7372.239202] ? do_raw_spin_unlock+0x75/0x190 [ 7372.241445] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7372.250156] ? do_wait_intr+0xf0/0xf0 [ 7372.256101] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7372.259374] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7372.262762] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7372.267489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7372.270163] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7372.277467] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7372.279343] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7372.288299] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7372.298441] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7372.305639] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7372.312441] lmv_intent_open+0x338/0x1150 [lmv] [ 7372.316309] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7372.321013] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7372.323406] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7372.325911] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7372.330030] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7372.333165] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7372.345474] ? do_raw_spin_unlock+0x75/0x190 [ 7372.347449] ? _raw_spin_unlock+0x12/0x30 [ 7372.348719] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7372.358621] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7372.366165] ll_atomic_open+0x36a/0xd70 [lustre] [ 7372.372040] lookup_open+0x356/0xb40 [ 7372.374657] do_last+0x313/0xfc0 [ 7372.377729] ? nd_jump_root+0xe5/0x160 [ 7372.380512] ? path_init+0x437/0x520 [ 7372.381480] path_openat+0xf7/0x500 [ 7372.382576] ? do_raw_spin_unlock+0x75/0x190 [ 7372.387440] do_filp_open+0x99/0x140 [ 7372.390742] ? getname_flags+0x6e/0x330 [ 7372.392802] ? __check_object_size+0xff/0x256 [ 7372.397911] ? do_raw_spin_unlock+0x75/0x190 [ 7372.401312] ? _raw_spin_unlock+0x12/0x30 [ 7372.403608] do_sys_openat2+0x2b4/0x410 [ 7372.405215] do_sys_open+0x73/0xa0 [ 7372.406715] __x64_sys_openat+0x24/0x30 [ 7372.408539] do_syscall_64+0xc1/0x440 [ 7372.410068] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7372.411937] RIP: 0033:0x7fdc0afbf332 [ 7372.414155] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7372.425641] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7372.433929] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7372.440475] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7372.445844] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7372.448948] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7372.454542] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7380.051090] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7380.055337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7380.098486] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7380.106822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7380.113180] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7380.117078] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7380.119148] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7380.130812] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7380.136557] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7380.143855] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7380.147176] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7380.156480] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7380.163838] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7380.174313] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7380.178184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7380.182159] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7380.188824] Call Trace: [ 7380.189930] [ 7380.190847] ? show_regs.cold.9+0x22/0x2f [ 7380.196875] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7380.199396] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7380.201888] ? watchdog+0x40/0x40 [ 7380.203082] ? hrtimer_interrupt+0x128/0x2c0 [ 7380.206992] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7380.209535] ? apic_timer_interrupt+0xf/0x20 [ 7380.214809] [ 7380.216299] ? native_safe_halt+0x2e/0x50 [ 7380.219780] kvm_wait+0xcd/0xe0 [ 7380.222147] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7380.224969] do_raw_spin_lock+0x177/0x190 [ 7380.228436] _raw_spin_lock+0x21/0x30 [ 7380.230758] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7380.234500] process_one_work+0x2c8/0x700 [ 7380.237922] worker_thread+0x296/0x6e0 [ 7380.239290] ? rescuer_thread+0x570/0x570 [ 7380.240932] kthread+0x1d1/0x200 [ 7380.243181] ? set_kthread_struct+0x70/0x70 [ 7380.244726] ret_from_fork+0x1f/0x30 [ 7388.058292] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7388.068099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7388.116682] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7388.125009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7388.135983] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7388.142017] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7388.154621] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7388.161851] RAX: 000000000000692f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7388.173979] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7388.186719] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7388.196017] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7388.205292] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7388.220495] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7388.232861] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7388.240830] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7388.250642] Call Trace: [ 7388.253039] [ 7388.256860] ? show_regs.cold.9+0x22/0x2f [ 7388.257923] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7388.261628] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7388.268775] ? watchdog+0x40/0x40 [ 7388.273736] ? hrtimer_interrupt+0x128/0x2c0 [ 7388.278872] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7388.285806] ? apic_timer_interrupt+0xf/0x20 [ 7388.290720] [ 7388.294993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7388.301796] do_raw_spin_lock+0x177/0x190 [ 7388.302943] _raw_spin_lock+0x21/0x30 [ 7388.307690] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7388.312362] ? do_raw_spin_unlock+0x75/0x190 [ 7388.317852] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7388.331086] ? do_wait_intr+0xf0/0xf0 [ 7388.334966] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7388.343872] mdc_close+0x53b/0xd70 [mdc] [ 7388.345802] lmv_close+0x18d/0x490 [lmv] [ 7388.348256] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7388.355950] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7388.358287] ll_md_close+0x274/0x800 [lustre] [ 7388.359686] ll_file_release+0x128/0xc50 [lustre] [ 7388.370854] __fput+0x10d/0x440 [ 7388.371814] ____fput+0x12/0x20 [ 7388.372773] task_work_run+0xc6/0x110 [ 7388.379428] exit_to_usermode_loop+0x1dd/0x1f0 [ 7388.383422] do_syscall_64+0x3d6/0x440 [ 7388.386752] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7388.392747] RIP: 0033:0x7fdc0afbfcd8 [ 7388.395736] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7388.416593] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7388.424701] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7388.431168] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7388.444277] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7388.446194] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7388.456890] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7392.048049] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7392.050945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7392.067622] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7392.071386] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7392.074730] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7392.077102] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7392.085499] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7392.088604] RAX: 0000000000005004 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7392.091630] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7392.094270] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7392.096397] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7392.098822] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7392.100963] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7392.103939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7392.105716] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7392.108202] Call Trace: [ 7392.108822] [ 7392.109516] ? show_regs.cold.9+0x22/0x2f [ 7392.111122] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7392.113345] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7392.115281] ? watchdog+0x40/0x40 [ 7392.116750] ? hrtimer_interrupt+0x128/0x2c0 [ 7392.118500] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7392.120641] ? apic_timer_interrupt+0xf/0x20 [ 7392.122815] [ 7392.124207] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7392.126839] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7392.129658] ? dump_stack+0xf6/0x10e [ 7392.131707] do_raw_spin_lock+0x177/0x190 [ 7392.133309] _raw_spin_lock+0x21/0x30 [ 7392.135013] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7392.137560] obd_mod_free+0x46/0x140 [ptlrpc] [ 7392.139527] mdc_commit_open+0x91/0xb0 [mdc] [ 7392.141547] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7392.143939] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7392.146631] after_reply+0xe2c/0x1780 [ptlrpc] [ 7392.148879] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7392.151621] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7392.153823] ? do_wait_intr+0xf0/0xf0 [ 7392.155684] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7392.158524] kthread+0x1d1/0x200 [ 7392.159973] ? set_kthread_struct+0x70/0x70 [ 7392.161629] ret_from_fork+0x1f/0x30 [ 7396.319197] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5371s! [ 7396.329026] Showing busy workqueues and worker pools: [ 7396.341861] workqueue events: flags=0x0 [ 7396.344283] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7396.351610] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7396.361749] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7396.369426] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7396.376187] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7396.386179] pending: kfree_rcu_monitor [ 7396.388162] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7396.395961] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7396.404437] workqueue events_long: flags=0x0 [ 7396.405995] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7396.416304] pending: lu_objects_flush [obdclass] [ 7396.420132] workqueue events_unbound: flags=0x2 [ 7396.428801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7396.431987] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7396.441821] workqueue events_power_efficient: flags=0x80 [ 7396.446570] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7396.454635] pending: neigh_periodic_work, check_lifetime [ 7396.463273] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7396.466796] pending: do_cache_clean, neigh_periodic_work [ 7396.472683] workqueue mm_percpu_wq: flags=0x8 [ 7396.477893] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7396.486725] pending: lru_add_drain_per_cpu [ 7396.488380] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7396.493817] pending: vmstat_update, lru_add_drain_per_cpu [ 7396.496701] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7396.503034] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7396.505799] workqueue writeback: flags=0x4a [ 7396.508442] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7396.512590] pending: wb_workfn, wb_workfn [ 7396.516626] workqueue xprtiod: flags=0xa [ 7396.520618] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7396.525231] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7396.533741] workqueue ipv6_addrconf: flags=0x40008 [ 7396.537928] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7396.544847] pending: addrconf_verify_work [ 7396.550316] workqueue knbd0-recv: flags=0x1a [ 7396.552498] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7396.559142] in-flight: 655:recv_work [ 7396.567143] workqueue ptlrpc_pinger: flags=0x2 [ 7396.571685] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7396.575683] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7396.579815] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5371s workers=4 idle: 36396 38 11188 [ 7396.588482] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7387s workers=2 idle: 658 [ 7400.053994] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7400.062723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7400.100638] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7400.105174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7400.110321] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7400.114831] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7400.125614] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7400.129002] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7400.133561] RDX: 0000000000004447 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7400.136852] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7400.141572] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7400.151899] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7400.158378] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7400.165765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7400.169899] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7400.177720] Call Trace: [ 7400.183880] [ 7400.184844] ? show_regs.cold.9+0x22/0x2f [ 7400.187668] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7400.191699] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7400.201764] ? watchdog+0x40/0x40 [ 7400.203481] ? hrtimer_interrupt+0x128/0x2c0 [ 7400.214959] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7400.219289] ? apic_timer_interrupt+0xf/0x20 [ 7400.221757] [ 7400.222811] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7400.225370] do_raw_spin_lock+0x177/0x190 [ 7400.232825] _raw_spin_lock+0x21/0x30 [ 7400.234641] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7400.247205] ? do_raw_spin_unlock+0x75/0x190 [ 7400.248696] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7400.259938] ? do_wait_intr+0xf0/0xf0 [ 7400.261286] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7400.277769] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7400.282113] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7400.288980] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7400.294992] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7400.300008] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7400.305395] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7400.311151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7400.314764] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7400.318040] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7400.326759] lmv_intent_open+0x338/0x1150 [lmv] [ 7400.333849] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7400.338203] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7400.343229] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7400.349459] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7400.354056] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7400.357828] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7400.360820] ? do_raw_spin_unlock+0x75/0x190 [ 7400.362497] ? _raw_spin_unlock+0x12/0x30 [ 7400.365095] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7400.367231] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7400.375977] ll_atomic_open+0x36a/0xd70 [lustre] [ 7400.379337] lookup_open+0x356/0xb40 [ 7400.380727] do_last+0x313/0xfc0 [ 7400.382035] ? nd_jump_root+0xe5/0x160 [ 7400.385170] ? path_init+0x437/0x520 [ 7400.386411] path_openat+0xf7/0x500 [ 7400.393100] ? do_raw_spin_unlock+0x75/0x190 [ 7400.394701] do_filp_open+0x99/0x140 [ 7400.395973] ? getname_flags+0x6e/0x330 [ 7400.399458] ? __check_object_size+0xff/0x256 [ 7400.405039] ? do_raw_spin_unlock+0x75/0x190 [ 7400.408760] ? _raw_spin_unlock+0x12/0x30 [ 7400.411466] do_sys_openat2+0x2b4/0x410 [ 7400.416706] do_sys_open+0x73/0xa0 [ 7400.419348] __x64_sys_openat+0x24/0x30 [ 7400.421776] do_syscall_64+0xc1/0x440 [ 7400.425061] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7400.427847] RIP: 0033:0x7fdc0afbf332 [ 7400.430343] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7400.444777] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7400.452356] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7400.459786] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7400.463374] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7400.470157] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7400.476814] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7408.051020] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7408.055890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7408.073626] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7408.078102] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7408.082820] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7408.086105] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7408.088764] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7408.097125] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7408.101124] RAX: 0000000000006e08 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7408.104109] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7408.107062] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7408.110041] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7408.113571] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7408.117060] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7408.120688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7408.123567] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7408.127699] Call Trace: [ 7408.129229] [ 7408.130204] ? show_regs.cold.9+0x22/0x2f [ 7408.131904] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7408.134091] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7408.135711] ? watchdog+0x40/0x40 [ 7408.136922] ? hrtimer_interrupt+0x128/0x2c0 [ 7408.138745] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7408.141366] ? apic_timer_interrupt+0xf/0x20 [ 7408.143528] [ 7408.144580] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7408.147284] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7408.150040] do_raw_spin_lock+0x177/0x190 [ 7408.152032] _raw_spin_lock+0x21/0x30 [ 7408.153681] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7408.156233] process_one_work+0x2c8/0x700 [ 7408.158151] worker_thread+0x296/0x6e0 [ 7408.159673] ? rescuer_thread+0x570/0x570 [ 7408.161581] kthread+0x1d1/0x200 [ 7408.163190] ? set_kthread_struct+0x70/0x70 [ 7408.165175] ret_from_fork+0x1f/0x30 [ 7416.057109] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7416.068567] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7416.100315] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7416.115135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7416.123523] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7416.126169] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7416.148738] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7416.154464] RAX: 0000000000001e17 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7416.163230] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7416.170924] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7416.176738] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7416.185893] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7416.193523] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7416.200353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7416.207329] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7416.212234] Call Trace: [ 7416.216990] [ 7416.217702] ? show_regs.cold.9+0x22/0x2f [ 7416.223526] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7416.231313] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7416.235924] ? watchdog+0x40/0x40 [ 7416.241449] ? hrtimer_interrupt+0x128/0x2c0 [ 7416.243292] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7416.245416] ? apic_timer_interrupt+0xf/0x20 [ 7416.247255] [ 7416.248129] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7416.250531] do_raw_spin_lock+0x177/0x190 [ 7416.252393] _raw_spin_lock+0x21/0x30 [ 7416.254304] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7416.257273] ? do_raw_spin_unlock+0x75/0x190 [ 7416.259222] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7416.261602] ? do_wait_intr+0xf0/0xf0 [ 7416.262890] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7416.265552] mdc_close+0x53b/0xd70 [mdc] [ 7416.267498] lmv_close+0x18d/0x490 [lmv] [ 7416.269272] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7416.272117] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7416.274500] ll_md_close+0x274/0x800 [lustre] [ 7416.276310] ll_file_release+0x128/0xc50 [lustre] [ 7416.279289] __fput+0x10d/0x440 [ 7416.280602] ____fput+0x12/0x20 [ 7416.283029] task_work_run+0xc6/0x110 [ 7416.284616] exit_to_usermode_loop+0x1dd/0x1f0 [ 7416.287362] do_syscall_64+0x3d6/0x440 [ 7416.288912] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7416.291525] RIP: 0033:0x7fdc0afbfcd8 [ 7416.293269] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7416.303381] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7416.307050] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7416.312463] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7416.315461] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7416.325621] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7416.330591] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7424.049256] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 7424.053314] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7424.079577] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7424.085916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7424.090959] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7424.094434] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7424.100642] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7424.103312] RAX: 0000000000004991 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7424.105910] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7424.108439] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7424.111463] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7424.114380] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7424.117333] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7424.121025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7424.123283] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7424.125988] Call Trace: [ 7424.126940] [ 7424.127672] ? show_regs.cold.9+0x22/0x2f [ 7424.129127] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7424.130728] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7424.132430] ? watchdog+0x40/0x40 [ 7424.133640] ? hrtimer_interrupt+0x128/0x2c0 [ 7424.135013] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7424.136657] ? apic_timer_interrupt+0xf/0x20 [ 7424.138244] [ 7424.139210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7424.141912] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7424.143434] ? dump_stack+0xf6/0x10e [ 7424.144977] do_raw_spin_lock+0x177/0x190 [ 7424.146855] _raw_spin_lock+0x21/0x30 [ 7424.148251] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7424.150549] obd_mod_free+0x46/0x140 [ptlrpc] [ 7424.152379] mdc_commit_open+0x91/0xb0 [mdc] [ 7424.154116] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7424.155908] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7424.158105] after_reply+0xe2c/0x1780 [ptlrpc] [ 7424.160262] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7424.163299] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7424.166861] ? do_wait_intr+0xf0/0xf0 [ 7424.169997] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7424.182841] kthread+0x1d1/0x200 [ 7424.184044] ? set_kthread_struct+0x70/0x70 [ 7424.187869] ret_from_fork+0x1f/0x30 [ 7427.039050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5402s! [ 7427.042028] Showing busy workqueues and worker pools: [ 7427.044665] workqueue events: flags=0x0 [ 7427.050227] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7427.055248] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7427.063051] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7427.067389] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7427.078359] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7427.089375] pending: kfree_rcu_monitor [ 7427.093084] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7427.097341] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7427.107077] workqueue events_long: flags=0x0 [ 7427.114105] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7427.117093] pending: lu_objects_flush [obdclass] [ 7427.119960] workqueue events_unbound: flags=0x2 [ 7427.126148] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7427.128484] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7427.139014] workqueue events_power_efficient: flags=0x80 [ 7427.142066] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7427.148778] pending: neigh_periodic_work, check_lifetime [ 7427.156975] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7427.162353] pending: do_cache_clean, neigh_periodic_work [ 7427.164512] workqueue mm_percpu_wq: flags=0x8 [ 7427.165592] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7427.175230] pending: lru_add_drain_per_cpu [ 7427.178317] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7427.186570] pending: vmstat_update, lru_add_drain_per_cpu [ 7427.192194] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7427.198610] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7427.203016] workqueue writeback: flags=0x4a [ 7427.206186] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7427.213113] pending: wb_workfn, wb_workfn [ 7427.216646] workqueue xprtiod: flags=0xa [ 7427.218895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7427.221949] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7427.231382] workqueue ipv6_addrconf: flags=0x40008 [ 7427.236954] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7427.240378] pending: addrconf_verify_work [ 7427.243016] workqueue knbd0-recv: flags=0x1a [ 7427.245231] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7427.249080] in-flight: 655:recv_work [ 7427.251993] workqueue ptlrpc_pinger: flags=0x2 [ 7427.254309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7427.258001] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7427.261980] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5402s workers=4 idle: 36396 38 11188 [ 7427.267348] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7417s workers=2 idle: 658 [ 7428.052055] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7428.055387] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7428.078140] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7428.084517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7428.088925] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7428.091174] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7428.099893] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7428.103145] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7428.105747] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7428.108393] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7428.111279] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7428.114550] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7428.117935] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7428.121816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7428.125592] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7428.129428] Call Trace: [ 7428.131416] [ 7428.132489] ? show_regs.cold.9+0x22/0x2f [ 7428.135262] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7428.138470] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7428.140652] ? watchdog+0x40/0x40 [ 7428.142651] ? hrtimer_interrupt+0x128/0x2c0 [ 7428.145198] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7428.147780] ? apic_timer_interrupt+0xf/0x20 [ 7428.150912] [ 7428.151966] ? native_safe_halt+0x2e/0x50 [ 7428.153936] kvm_wait+0xcd/0xe0 [ 7428.154952] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7428.157503] do_raw_spin_lock+0x177/0x190 [ 7428.159348] _raw_spin_lock+0x21/0x30 [ 7428.161354] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7428.164243] ? do_raw_spin_unlock+0x75/0x190 [ 7428.166352] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7428.169494] ? do_wait_intr+0xf0/0xf0 [ 7428.171217] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7428.173972] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7428.176181] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7428.179032] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7428.181682] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7428.184251] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7428.186669] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7428.189218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7428.191964] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7428.195509] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7428.198178] lmv_intent_open+0x338/0x1150 [lmv] [ 7428.200126] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7428.203432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7428.206371] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7428.208406] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7428.211178] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7428.213744] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7428.216282] ? do_raw_spin_unlock+0x75/0x190 [ 7428.218143] ? _raw_spin_unlock+0x12/0x30 [ 7428.220216] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7428.222683] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7428.225464] ll_atomic_open+0x36a/0xd70 [lustre] [ 7428.228235] lookup_open+0x356/0xb40 [ 7428.229982] do_last+0x313/0xfc0 [ 7428.232117] ? nd_jump_root+0xe5/0x160 [ 7428.233931] ? path_init+0x437/0x520 [ 7428.235731] path_openat+0xf7/0x500 [ 7428.237493] ? do_raw_spin_unlock+0x75/0x190 [ 7428.239733] do_filp_open+0x99/0x140 [ 7428.241815] ? getname_flags+0x6e/0x330 [ 7428.243581] ? __check_object_size+0xff/0x256 [ 7428.246433] ? do_raw_spin_unlock+0x75/0x190 [ 7428.248654] ? _raw_spin_unlock+0x12/0x30 [ 7428.250599] do_sys_openat2+0x2b4/0x410 [ 7428.252253] do_sys_open+0x73/0xa0 [ 7428.253912] __x64_sys_openat+0x24/0x30 [ 7428.256479] do_syscall_64+0xc1/0x440 [ 7428.258353] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7428.260670] RIP: 0033:0x7fdc0afbf332 [ 7428.262504] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7428.270983] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7428.274450] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7428.277599] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7428.280570] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7428.283846] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7428.287078] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7436.051029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7436.054147] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7436.101743] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7436.107832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7436.111582] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7436.113874] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7436.115516] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7436.124788] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7436.127008] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7436.129516] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7436.131626] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7436.133558] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7436.135468] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7436.138000] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7436.141347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7436.143579] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7436.147957] Call Trace: [ 7436.148618] [ 7436.149454] ? show_regs.cold.9+0x22/0x2f [ 7436.151914] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7436.153549] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7436.155256] ? watchdog+0x40/0x40 [ 7436.156306] ? hrtimer_interrupt+0x128/0x2c0 [ 7436.157822] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7436.160166] ? apic_timer_interrupt+0xf/0x20 [ 7436.162107] [ 7436.163228] ? native_safe_halt+0x2e/0x50 [ 7436.164905] kvm_wait+0xcd/0xe0 [ 7436.165948] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7436.167601] do_raw_spin_lock+0x177/0x190 [ 7436.169045] _raw_spin_lock+0x21/0x30 [ 7436.173550] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7436.176199] process_one_work+0x2c8/0x700 [ 7436.180659] worker_thread+0x296/0x6e0 [ 7436.182154] ? rescuer_thread+0x570/0x570 [ 7436.184058] kthread+0x1d1/0x200 [ 7436.185461] ? set_kthread_struct+0x70/0x70 [ 7436.187109] ret_from_fork+0x1f/0x30 [ 7444.054028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7444.061818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7444.110040] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7444.115892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7444.120357] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7444.122593] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7444.130803] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7444.133350] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7444.135941] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7444.138869] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7444.142928] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7444.146358] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7444.149624] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7444.153875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7444.156845] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7444.160389] Call Trace: [ 7444.161557] [ 7444.162163] ? show_regs.cold.9+0x22/0x2f [ 7444.163499] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7444.166821] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7444.169625] ? watchdog+0x40/0x40 [ 7444.172238] ? hrtimer_interrupt+0x128/0x2c0 [ 7444.177255] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7444.181309] ? apic_timer_interrupt+0xf/0x20 [ 7444.184253] [ 7444.185819] ? native_safe_halt+0x2e/0x50 [ 7444.188232] kvm_wait+0xcd/0xe0 [ 7444.189960] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7444.193559] do_raw_spin_lock+0x177/0x190 [ 7444.195696] _raw_spin_lock+0x21/0x30 [ 7444.197721] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7444.200895] ? do_raw_spin_unlock+0x75/0x190 [ 7444.203503] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7444.206495] ? do_wait_intr+0xf0/0xf0 [ 7444.208348] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7444.211822] mdc_close+0x53b/0xd70 [mdc] [ 7444.214008] lmv_close+0x18d/0x490 [lmv] [ 7444.216476] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7444.218591] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7444.221831] ll_md_close+0x274/0x800 [lustre] [ 7444.224600] ll_file_release+0x128/0xc50 [lustre] [ 7444.227101] __fput+0x10d/0x440 [ 7444.228916] ____fput+0x12/0x20 [ 7444.230905] task_work_run+0xc6/0x110 [ 7444.232712] exit_to_usermode_loop+0x1dd/0x1f0 [ 7444.236045] do_syscall_64+0x3d6/0x440 [ 7444.238626] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7444.241268] RIP: 0033:0x7fdc0afbfcd8 [ 7444.243862] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7444.254075] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7444.258620] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7444.262225] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7444.265376] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7444.267889] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7444.270622] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7452.049110] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 7452.054031] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7452.115574] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7452.121552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7452.127546] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7452.137404] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7452.149091] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7452.153904] RAX: 0000000000006273 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7452.156421] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7452.162571] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7452.171275] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7452.174498] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7452.178485] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7452.182526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7452.184932] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7452.190398] Call Trace: [ 7452.192214] [ 7452.193471] ? show_regs.cold.9+0x22/0x2f [ 7452.198825] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7452.201631] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7452.204730] ? watchdog+0x40/0x40 [ 7452.205795] ? hrtimer_interrupt+0x128/0x2c0 [ 7452.213734] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7452.219577] ? apic_timer_interrupt+0xf/0x20 [ 7452.223120] [ 7452.226744] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7452.229864] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7452.234800] ? dump_stack+0xf6/0x10e [ 7452.238463] do_raw_spin_lock+0x177/0x190 [ 7452.241755] _raw_spin_lock+0x21/0x30 [ 7452.243820] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7452.246654] obd_mod_free+0x46/0x140 [ptlrpc] [ 7452.249271] mdc_commit_open+0x91/0xb0 [mdc] [ 7452.251516] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7452.255352] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7452.259538] after_reply+0xe2c/0x1780 [ptlrpc] [ 7452.263336] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7452.266257] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7452.272598] ? do_wait_intr+0xf0/0xf0 [ 7452.274037] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7452.287080] kthread+0x1d1/0x200 [ 7452.288546] ? set_kthread_struct+0x70/0x70 [ 7452.292271] ret_from_fork+0x1f/0x30 [ 7456.052007] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7456.055983] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7456.075839] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7456.081595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7456.086753] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7456.092330] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7456.101771] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7456.104455] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7456.110781] RDX: 0000000000002ece RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7456.116488] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7456.119428] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7456.128142] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7456.130253] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7456.133566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7456.135877] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7456.138380] Call Trace: [ 7456.139347] [ 7456.140096] ? show_regs.cold.9+0x22/0x2f [ 7456.141790] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7456.144859] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7456.147799] ? watchdog+0x40/0x40 [ 7456.149788] ? hrtimer_interrupt+0x128/0x2c0 [ 7456.152481] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7456.154846] ? apic_timer_interrupt+0xf/0x20 [ 7456.156454] [ 7456.157190] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7456.162065] do_raw_spin_lock+0x177/0x190 [ 7456.165800] _raw_spin_lock+0x21/0x30 [ 7456.170162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7456.175467] ? do_raw_spin_unlock+0x75/0x190 [ 7456.176930] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7456.182682] ? do_wait_intr+0xf0/0xf0 [ 7456.183532] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7456.185165] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7456.187503] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7456.189889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7456.191973] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7456.194117] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7456.195862] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7456.197526] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7456.200063] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7456.206831] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7456.214089] lmv_intent_open+0x338/0x1150 [lmv] [ 7456.219417] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7456.226463] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7456.228807] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7456.230617] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7456.234595] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7456.238535] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7456.244291] ? do_raw_spin_unlock+0x75/0x190 [ 7456.248374] ? _raw_spin_unlock+0x12/0x30 [ 7456.250009] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7456.253401] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7456.255644] ll_atomic_open+0x36a/0xd70 [lustre] [ 7456.258626] lookup_open+0x356/0xb40 [ 7456.264027] do_last+0x313/0xfc0 [ 7456.266634] ? nd_jump_root+0xe5/0x160 [ 7456.271489] ? path_init+0x437/0x520 [ 7456.276113] path_openat+0xf7/0x500 [ 7456.278376] ? do_raw_spin_unlock+0x75/0x190 [ 7456.280422] do_filp_open+0x99/0x140 [ 7456.284544] ? getname_flags+0x6e/0x330 [ 7456.286605] ? __check_object_size+0xff/0x256 [ 7456.289479] ? do_raw_spin_unlock+0x75/0x190 [ 7456.291655] ? _raw_spin_unlock+0x12/0x30 [ 7456.293049] do_sys_openat2+0x2b4/0x410 [ 7456.296476] do_sys_open+0x73/0xa0 [ 7456.297684] __x64_sys_openat+0x24/0x30 [ 7456.299067] do_syscall_64+0xc1/0x440 [ 7456.303640] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7456.307459] RIP: 0033:0x7fdc0afbf332 [ 7456.309136] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7456.322282] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7456.325488] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7456.328545] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7456.330874] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7456.336514] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7456.342497] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7457.759081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5432s! [ 7457.763478] Showing busy workqueues and worker pools: [ 7457.766906] workqueue events: flags=0x0 [ 7457.768398] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7457.772973] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7457.777104] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7457.781301] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7457.787339] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7457.794813] pending: kfree_rcu_monitor [ 7457.800776] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7457.808095] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7457.819920] workqueue events_long: flags=0x0 [ 7457.823170] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7457.827472] pending: lu_objects_flush [obdclass] [ 7457.837454] workqueue events_unbound: flags=0x2 [ 7457.844118] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7457.848658] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7457.859847] workqueue events_power_efficient: flags=0x80 [ 7457.863904] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7457.867852] pending: neigh_periodic_work, check_lifetime [ 7457.871268] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7457.875346] pending: do_cache_clean, neigh_periodic_work [ 7457.877742] workqueue mm_percpu_wq: flags=0x8 [ 7457.879720] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7457.883148] pending: lru_add_drain_per_cpu [ 7457.886594] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7457.892799] pending: vmstat_update, lru_add_drain_per_cpu [ 7457.897331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7457.902923] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7457.906255] workqueue writeback: flags=0x4a [ 7457.908175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7457.913796] pending: wb_workfn, wb_workfn [ 7457.916744] workqueue xprtiod: flags=0xa [ 7457.919810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7457.923678] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7457.928748] workqueue ipv6_addrconf: flags=0x40008 [ 7457.932274] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7457.937380] pending: addrconf_verify_work [ 7457.942098] workqueue knbd0-recv: flags=0x1a [ 7457.946193] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7457.953338] in-flight: 655:recv_work [ 7457.956040] workqueue ptlrpc_pinger: flags=0x2 [ 7457.959166] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7457.962124] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7457.966686] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5433s workers=4 idle: 36396 38 11188 [ 7457.973826] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7448s workers=2 idle: 658 [ 7464.051032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7464.053729] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7464.104382] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7464.113682] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7464.117313] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7464.120373] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7464.122306] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7464.129939] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7464.133107] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7464.140351] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7464.149107] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7464.162072] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7464.165214] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7464.169350] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7464.172898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7464.177620] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7464.181742] Call Trace: [ 7464.187445] [ 7464.188372] ? show_regs.cold.9+0x22/0x2f [ 7464.192383] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7464.195161] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7464.196712] ? watchdog+0x40/0x40 [ 7464.197863] ? hrtimer_interrupt+0x128/0x2c0 [ 7464.200060] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7464.202125] ? apic_timer_interrupt+0xf/0x20 [ 7464.203930] [ 7464.204913] ? native_safe_halt+0x2e/0x50 [ 7464.206498] kvm_wait+0xcd/0xe0 [ 7464.207913] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7464.210039] do_raw_spin_lock+0x177/0x190 [ 7464.211150] _raw_spin_lock+0x21/0x30 [ 7464.212682] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7464.215221] process_one_work+0x2c8/0x700 [ 7464.216540] worker_thread+0x296/0x6e0 [ 7464.218245] ? rescuer_thread+0x570/0x570 [ 7464.219932] kthread+0x1d1/0x200 [ 7464.221929] ? set_kthread_struct+0x70/0x70 [ 7464.223424] ret_from_fork+0x1f/0x30 [ 7472.056530] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7472.065547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7472.098401] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7472.110388] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7472.119150] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7472.123351] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7472.136020] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7472.143189] RAX: 0000000000001b91 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7472.151632] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7472.159858] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7472.164576] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7472.169338] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7472.175497] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7472.179916] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7472.182395] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7472.186178] Call Trace: [ 7472.187937] [ 7472.188869] ? show_regs.cold.9+0x22/0x2f [ 7472.190479] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7472.192509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7472.194115] ? watchdog+0x40/0x40 [ 7472.195548] ? hrtimer_interrupt+0x128/0x2c0 [ 7472.197843] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7472.200306] ? apic_timer_interrupt+0xf/0x20 [ 7472.202431] [ 7472.203814] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7472.209454] do_raw_spin_lock+0x177/0x190 [ 7472.210599] _raw_spin_lock+0x21/0x30 [ 7472.211726] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7472.216990] ? do_raw_spin_unlock+0x75/0x190 [ 7472.218887] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7472.221205] ? do_wait_intr+0xf0/0xf0 [ 7472.222916] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7472.225501] mdc_close+0x53b/0xd70 [mdc] [ 7472.228104] lmv_close+0x18d/0x490 [lmv] [ 7472.235438] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7472.238354] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7472.241779] ll_md_close+0x274/0x800 [lustre] [ 7472.244432] ll_file_release+0x128/0xc50 [lustre] [ 7472.247744] __fput+0x10d/0x440 [ 7472.249403] ____fput+0x12/0x20 [ 7472.250616] task_work_run+0xc6/0x110 [ 7472.252055] exit_to_usermode_loop+0x1dd/0x1f0 [ 7472.255828] do_syscall_64+0x3d6/0x440 [ 7472.259356] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7472.264758] RIP: 0033:0x7fdc0afbfcd8 [ 7472.266599] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7472.278498] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7472.284359] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7472.289345] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7472.292099] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7472.298110] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7472.304357] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7479.481853] rcu: INFO: rcu_sched self-detected stall on CPU [ 7479.484315] rcu: 0-...!: (3365122 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 7479.490671] (t=5460138 jiffies g=270761 q=3970) [ 7479.497465] rcu: rcu_sched kthread starved for 5455070 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 7479.513511] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7479.523629] rcu: RCU grace-period kthread stack dump: [ 7479.526721] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7479.532688] Call Trace: [ 7479.542557] __schedule+0x351/0xcb0 [ 7479.545161] schedule+0xc0/0x180 [ 7479.555602] schedule_timeout+0xb4/0x190 [ 7479.570115] ? do_raw_spin_unlock+0x75/0x190 [ 7479.578973] ? __next_timer_interrupt+0x160/0x160 [ 7479.580650] ? prepare_to_swait_event+0xf3/0x140 [ 7479.603777] rcu_gp_kthread+0x8dd/0x15b0 [ 7479.605241] ? force_qs_rnp+0x2c0/0x2c0 [ 7479.615896] kthread+0x1d1/0x200 [ 7479.623065] ? set_kthread_struct+0x70/0x70 [ 7479.627283] ret_from_fork+0x1f/0x30 [ 7479.632484] rcu: Stack dump where RCU GP kthread last ran: [ 7479.640945] NMI backtrace for cpu 0 [ 7479.647607] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7479.658587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7479.665041] Call Trace: [ 7479.670283] [ 7479.672621] ? dump_stack+0xbb/0x10e [ 7479.677393] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7479.686841] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7479.693085] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7479.700443] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7479.705138] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7479.711371] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 7479.717393] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 7479.720460] ? update_process_times+0x6d/0xc0 [ 7479.722477] ? tick_sched_handle.isra.18+0x27/0x90 [ 7479.726205] ? tick_sched_timer+0x53/0xb0 [ 7479.728685] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7479.735401] ? tick_sched_handle.isra.18+0x90/0x90 [ 7479.745751] ? hrtimer_interrupt+0x128/0x2c0 [ 7479.752402] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7479.757492] ? apic_timer_interrupt+0xf/0x20 [ 7479.770367] [ 7479.781047] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7479.795641] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7479.804319] ? dump_stack+0xf6/0x10e [ 7479.807759] ? do_raw_spin_lock+0x177/0x190 [ 7479.813391] ? _raw_spin_lock+0x21/0x30 [ 7479.824252] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7479.844708] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7479.858190] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7479.867103] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7479.872930] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7479.876932] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7479.881968] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7479.892848] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7479.900981] ? do_wait_intr+0xf0/0xf0 [ 7479.909180] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7479.911827] ? kthread+0x1d1/0x200 [ 7479.921091] ? set_kthread_struct+0x70/0x70 [ 7479.930182] ? ret_from_fork+0x1f/0x30 [ 7479.935637] NMI backtrace for cpu 0 [ 7479.939188] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7479.958196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7479.973885] Call Trace: [ 7479.976027] [ 7479.980245] ? dump_stack+0xbb/0x10e [ 7479.981990] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7479.993155] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7480.003817] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7480.010698] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7480.017073] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7480.022650] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7480.028357] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7480.035223] ? update_process_times+0x6d/0xc0 [ 7480.044146] ? tick_sched_handle.isra.18+0x27/0x90 [ 7480.052654] ? tick_sched_timer+0x53/0xb0 [ 7480.058480] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7480.063932] ? tick_sched_handle.isra.18+0x90/0x90 [ 7480.072894] ? hrtimer_interrupt+0x128/0x2c0 [ 7480.079171] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7480.084784] ? apic_timer_interrupt+0xf/0x20 [ 7480.089890] [ 7480.092994] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7480.096102] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7480.103068] ? dump_stack+0xf6/0x10e [ 7480.113448] ? do_raw_spin_lock+0x177/0x190 [ 7480.118945] ? _raw_spin_lock+0x21/0x30 [ 7480.126634] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7480.140876] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7480.149319] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7480.155208] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7480.161036] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7480.170217] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7480.175846] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7480.186020] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7480.191928] ? do_wait_intr+0xf0/0xf0 [ 7480.194919] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7480.199378] ? kthread+0x1d1/0x200 [ 7480.202559] ? set_kthread_struct+0x70/0x70 [ 7480.211469] ? ret_from_fork+0x1f/0x30 [ 7484.054009] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 7484.064884] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7484.096727] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7484.103709] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7484.108982] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7484.113025] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7484.126996] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7484.132945] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7484.137925] RDX: 0000000000004664 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7484.145772] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7484.150513] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7484.155650] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7484.160359] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7484.166370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7484.170464] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7484.174996] Call Trace: [ 7484.179955] [ 7484.181711] ? show_regs.cold.9+0x22/0x2f [ 7484.185708] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7484.188149] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7484.190537] ? watchdog+0x40/0x40 [ 7484.192273] ? hrtimer_interrupt+0x128/0x2c0 [ 7484.194737] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7484.196909] ? apic_timer_interrupt+0xf/0x20 [ 7484.198933] [ 7484.200090] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7484.202877] do_raw_spin_lock+0x177/0x190 [ 7484.204780] _raw_spin_lock+0x21/0x30 [ 7484.206743] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7484.209600] ? do_raw_spin_unlock+0x75/0x190 [ 7484.211898] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7484.214622] ? do_wait_intr+0xf0/0xf0 [ 7484.216555] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7484.219156] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7484.222389] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7484.225589] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.228576] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7484.231494] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7484.233757] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7484.235850] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.238689] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7484.242257] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7484.245378] lmv_intent_open+0x338/0x1150 [lmv] [ 7484.247870] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.250803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.253972] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7484.256816] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7484.258841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.262390] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7484.265034] ? do_raw_spin_unlock+0x75/0x190 [ 7484.268149] ? _raw_spin_unlock+0x12/0x30 [ 7484.271833] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7484.274463] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7484.277393] ll_atomic_open+0x36a/0xd70 [lustre] [ 7484.279971] lookup_open+0x356/0xb40 [ 7484.282612] do_last+0x313/0xfc0 [ 7484.284280] ? nd_jump_root+0xe5/0x160 [ 7484.286612] ? path_init+0x437/0x520 [ 7484.288883] path_openat+0xf7/0x500 [ 7484.291174] ? do_raw_spin_unlock+0x75/0x190 [ 7484.293782] do_filp_open+0x99/0x140 [ 7484.295743] ? getname_flags+0x6e/0x330 [ 7484.297841] ? __check_object_size+0xff/0x256 [ 7484.299218] ? do_raw_spin_unlock+0x75/0x190 [ 7484.300723] ? _raw_spin_unlock+0x12/0x30 [ 7484.302753] do_sys_openat2+0x2b4/0x410 [ 7484.304855] do_sys_open+0x73/0xa0 [ 7484.306791] __x64_sys_openat+0x24/0x30 [ 7484.309375] do_syscall_64+0xc1/0x440 [ 7484.311088] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7484.312875] RIP: 0033:0x7fdc0afbf332 [ 7484.314108] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7484.324840] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7484.328732] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7484.331960] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7484.335555] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7484.339280] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7484.343497] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7488.479045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5463s! [ 7488.488826] Showing busy workqueues and worker pools: [ 7488.490283] workqueue events: flags=0x0 [ 7488.491546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7488.497143] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7488.500472] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7488.503212] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7488.506953] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7488.509568] pending: kfree_rcu_monitor [ 7488.511416] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7488.515971] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7488.528267] workqueue events_long: flags=0x0 [ 7488.530808] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7488.533955] pending: lu_objects_flush [obdclass] [ 7488.539852] workqueue events_unbound: flags=0x2 [ 7488.542256] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7488.545769] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7488.549735] workqueue events_power_efficient: flags=0x80 [ 7488.551370] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7488.555912] pending: neigh_periodic_work, check_lifetime [ 7488.560096] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7488.563408] pending: do_cache_clean, neigh_periodic_work [ 7488.567204] workqueue mm_percpu_wq: flags=0x8 [ 7488.568878] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7488.574429] pending: lru_add_drain_per_cpu [ 7488.576975] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7488.582917] pending: vmstat_update, lru_add_drain_per_cpu [ 7488.586028] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7488.590011] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7488.596421] workqueue writeback: flags=0x4a [ 7488.598878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7488.606569] pending: wb_workfn, wb_workfn [ 7488.613395] workqueue xprtiod: flags=0xa [ 7488.617914] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7488.620918] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7488.627339] workqueue ipv6_addrconf: flags=0x40008 [ 7488.629533] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7488.631550] pending: addrconf_verify_work [ 7488.633565] workqueue knbd0-recv: flags=0x1a [ 7488.635317] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7488.641613] in-flight: 655:recv_work [ 7488.645155] workqueue ptlrpc_pinger: flags=0x2 [ 7488.647022] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7488.651790] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7488.657208] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5463s workers=4 idle: 36396 38 11188 [ 7488.661509] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7479s workers=2 idle: 658 [ 7492.051028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7492.054132] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7492.078156] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7492.084043] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7492.089911] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7492.095243] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7492.097375] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7492.106754] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7492.110566] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7492.114186] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7492.117318] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7492.119886] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7492.122181] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7492.125193] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7492.128434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7492.130451] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7492.133172] Call Trace: [ 7492.134125] [ 7492.134872] ? show_regs.cold.9+0x22/0x2f [ 7492.136176] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7492.138160] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7492.139887] ? watchdog+0x40/0x40 [ 7492.140995] ? hrtimer_interrupt+0x128/0x2c0 [ 7492.142432] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7492.144538] ? apic_timer_interrupt+0xf/0x20 [ 7492.146193] [ 7492.147048] ? native_safe_halt+0x2e/0x50 [ 7492.148464] kvm_wait+0xcd/0xe0 [ 7492.149823] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7492.151781] do_raw_spin_lock+0x177/0x190 [ 7492.152988] _raw_spin_lock+0x21/0x30 [ 7492.154466] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7492.156890] process_one_work+0x2c8/0x700 [ 7492.162008] worker_thread+0x296/0x6e0 [ 7492.164116] ? rescuer_thread+0x570/0x570 [ 7492.167057] kthread+0x1d1/0x200 [ 7492.169238] ? set_kthread_struct+0x70/0x70 [ 7492.171780] ret_from_fork+0x1f/0x30 [ 7500.055646] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7500.060485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7500.094698] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7500.110752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7500.123416] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7500.128385] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7500.150087] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7500.158480] RAX: 0000000000006413 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7500.167760] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7500.181858] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7500.187436] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7500.191660] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7500.195187] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7500.205450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7500.207862] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7500.214092] Call Trace: [ 7500.217532] [ 7500.218915] ? show_regs.cold.9+0x22/0x2f [ 7500.220915] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7500.222984] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7500.225003] ? watchdog+0x40/0x40 [ 7500.230415] ? hrtimer_interrupt+0x128/0x2c0 [ 7500.234926] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7500.242779] ? apic_timer_interrupt+0xf/0x20 [ 7500.247482] [ 7500.249343] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7500.254905] do_raw_spin_lock+0x177/0x190 [ 7500.257746] _raw_spin_lock+0x21/0x30 [ 7500.260040] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7500.262247] ? do_raw_spin_unlock+0x75/0x190 [ 7500.264347] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7500.267375] ? do_wait_intr+0xf0/0xf0 [ 7500.269078] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7500.272023] mdc_close+0x53b/0xd70 [mdc] [ 7500.274056] lmv_close+0x18d/0x490 [lmv] [ 7500.275966] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7500.280364] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7500.284053] ll_md_close+0x274/0x800 [lustre] [ 7500.288570] ll_file_release+0x128/0xc50 [lustre] [ 7500.291478] __fput+0x10d/0x440 [ 7500.293005] ____fput+0x12/0x20 [ 7500.294343] task_work_run+0xc6/0x110 [ 7500.296078] exit_to_usermode_loop+0x1dd/0x1f0 [ 7500.298293] do_syscall_64+0x3d6/0x440 [ 7500.300034] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7500.302822] RIP: 0033:0x7fdc0afbfcd8 [ 7500.304374] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7500.312980] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7500.316057] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7500.318276] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7500.323257] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7500.327697] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7500.331148] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7512.052073] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 7512.056827] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7512.088179] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7512.094544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7512.099649] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7512.102878] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7512.116525] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7512.122158] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7512.127132] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7512.130895] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7512.135724] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7512.139821] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7512.144391] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7512.149600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7512.153105] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7512.157530] Call Trace: [ 7512.158913] [ 7512.160102] ? show_regs.cold.9+0x22/0x2f [ 7512.162621] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7512.165511] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7512.167948] ? watchdog+0x40/0x40 [ 7512.171195] ? hrtimer_interrupt+0x128/0x2c0 [ 7512.173351] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7512.176094] ? apic_timer_interrupt+0xf/0x20 [ 7512.178807] [ 7512.180174] ? native_safe_halt+0x2e/0x50 [ 7512.182640] kvm_wait+0xcd/0xe0 [ 7512.185194] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7512.188415] do_raw_spin_lock+0x177/0x190 [ 7512.190900] _raw_spin_lock+0x21/0x30 [ 7512.193223] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7512.196903] ? do_raw_spin_unlock+0x75/0x190 [ 7512.199544] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7512.202611] ? do_wait_intr+0xf0/0xf0 [ 7512.205073] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7512.208301] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7512.211926] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7512.215718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7512.219204] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7512.222136] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7512.224887] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7512.228657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7512.232133] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7512.235574] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7512.239383] lmv_intent_open+0x338/0x1150 [lmv] [ 7512.242531] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7512.245412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7512.248476] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7512.250282] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7512.252512] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7512.255261] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7512.258088] ? do_raw_spin_unlock+0x75/0x190 [ 7512.259443] ? _raw_spin_unlock+0x12/0x30 [ 7512.261404] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7512.264370] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7512.266990] ll_atomic_open+0x36a/0xd70 [lustre] [ 7512.269498] lookup_open+0x356/0xb40 [ 7512.271361] do_last+0x313/0xfc0 [ 7512.272771] ? nd_jump_root+0xe5/0x160 [ 7512.274599] ? path_init+0x437/0x520 [ 7512.275963] path_openat+0xf7/0x500 [ 7512.278129] ? do_raw_spin_unlock+0x75/0x190 [ 7512.280558] do_filp_open+0x99/0x140 [ 7512.282864] ? getname_flags+0x6e/0x330 [ 7512.286065] ? __check_object_size+0xff/0x256 [ 7512.289884] ? do_raw_spin_unlock+0x75/0x190 [ 7512.292072] ? _raw_spin_unlock+0x12/0x30 [ 7512.298549] do_sys_openat2+0x2b4/0x410 [ 7512.301041] do_sys_open+0x73/0xa0 [ 7512.303230] __x64_sys_openat+0x24/0x30 [ 7512.305127] do_syscall_64+0xc1/0x440 [ 7512.306656] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7512.309730] RIP: 0033:0x7fdc0afbf332 [ 7512.313275] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7512.324393] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7512.330019] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7512.335131] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7512.339895] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7512.344488] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7512.347980] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7516.050768] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7516.058993] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7516.086286] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7516.091739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7516.094901] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7516.096619] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7516.102409] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7516.105620] RAX: 0000000000007e0d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7516.107771] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7516.110369] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7516.112991] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7516.114982] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7516.117079] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7516.119705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7516.122084] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7516.124578] Call Trace: [ 7516.125223] [ 7516.125812] ? show_regs.cold.9+0x22/0x2f [ 7516.127176] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7516.128956] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7516.130696] ? watchdog+0x40/0x40 [ 7516.131783] ? hrtimer_interrupt+0x128/0x2c0 [ 7516.133028] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7516.134306] ? apic_timer_interrupt+0xf/0x20 [ 7516.135531] [ 7516.136150] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7516.137719] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7516.139405] ? dump_stack+0xf6/0x10e [ 7516.140858] do_raw_spin_lock+0x177/0x190 [ 7516.142393] _raw_spin_lock+0x21/0x30 [ 7516.143923] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7516.145620] obd_mod_free+0x46/0x140 [ptlrpc] [ 7516.147115] mdc_commit_open+0x91/0xb0 [mdc] [ 7516.148373] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7516.150646] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7516.152906] after_reply+0xe2c/0x1780 [ptlrpc] [ 7516.154375] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7516.155876] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7516.157152] ? do_wait_intr+0xf0/0xf0 [ 7516.158743] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7516.160417] kthread+0x1d1/0x200 [ 7516.161783] ? set_kthread_struct+0x70/0x70 [ 7516.163613] ret_from_fork+0x1f/0x30 [ 7518.687090] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5493s! [ 7518.692214] Showing busy workqueues and worker pools: [ 7518.695588] workqueue events: flags=0x0 [ 7518.697667] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7518.701442] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7518.705294] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7518.709003] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7518.716321] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7518.719266] pending: kfree_rcu_monitor [ 7518.721037] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7518.724353] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7518.731437] workqueue events_long: flags=0x0 [ 7518.733475] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7518.736477] pending: lu_objects_flush [obdclass] [ 7518.738440] workqueue events_unbound: flags=0x2 [ 7518.740288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7518.743251] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7518.749233] workqueue events_power_efficient: flags=0x80 [ 7518.751615] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7518.754604] pending: neigh_periodic_work, check_lifetime [ 7518.757172] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7518.760895] pending: do_cache_clean, neigh_periodic_work [ 7518.762839] workqueue mm_percpu_wq: flags=0x8 [ 7518.764471] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7518.767204] pending: lru_add_drain_per_cpu [ 7518.768893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7518.771659] pending: vmstat_update, lru_add_drain_per_cpu [ 7518.773986] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7518.776770] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7518.779594] workqueue writeback: flags=0x4a [ 7518.781309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7518.784353] pending: wb_workfn, wb_workfn [ 7518.786148] workqueue xprtiod: flags=0xa [ 7518.787919] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7518.790458] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7518.793936] workqueue ipv6_addrconf: flags=0x40008 [ 7518.795916] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7518.798695] pending: addrconf_verify_work [ 7518.800903] workqueue knbd0-recv: flags=0x1a [ 7518.803477] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7518.806367] in-flight: 655:recv_work [ 7518.808823] workqueue ptlrpc_pinger: flags=0x2 [ 7518.810552] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7518.812252] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7518.814606] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5493s workers=4 idle: 36396 38 11188 [ 7518.818046] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7509s workers=2 idle: 658 [ 7520.051029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7520.056575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7520.083643] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7520.091162] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7520.095477] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7520.098421] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7520.100227] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7520.110155] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7520.113425] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7520.116265] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7520.123438] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7520.130637] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7520.132387] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7520.134328] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7520.138435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7520.141373] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7520.143096] Call Trace: [ 7520.144747] [ 7520.145653] ? show_regs.cold.9+0x22/0x2f [ 7520.147770] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7520.149769] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7520.151737] ? watchdog+0x40/0x40 [ 7520.154406] ? hrtimer_interrupt+0x128/0x2c0 [ 7520.158333] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7520.161370] ? apic_timer_interrupt+0xf/0x20 [ 7520.162856] [ 7520.165460] ? native_safe_halt+0x2e/0x50 [ 7520.168375] kvm_wait+0xcd/0xe0 [ 7520.170507] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7520.174375] do_raw_spin_lock+0x177/0x190 [ 7520.176292] _raw_spin_lock+0x21/0x30 [ 7520.180286] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7520.183123] process_one_work+0x2c8/0x700 [ 7520.186665] worker_thread+0x296/0x6e0 [ 7520.188478] ? rescuer_thread+0x570/0x570 [ 7520.190953] kthread+0x1d1/0x200 [ 7520.192286] ? set_kthread_struct+0x70/0x70 [ 7520.194879] ret_from_fork+0x1f/0x30 [ 7528.058405] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 7528.061041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7528.089788] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7528.094597] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7528.099037] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7528.101547] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7528.112347] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7528.114837] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7528.118113] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7528.124972] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7528.130312] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7528.133593] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7528.135892] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7528.138275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7528.139967] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7528.142139] Call Trace: [ 7528.142883] [ 7528.143458] ? show_regs.cold.9+0x22/0x2f [ 7528.145034] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7528.146449] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7528.148185] ? watchdog+0x40/0x40 [ 7528.149300] ? hrtimer_interrupt+0x128/0x2c0 [ 7528.150983] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7528.152599] ? apic_timer_interrupt+0xf/0x20 [ 7528.154110] [ 7528.154738] ? native_safe_halt+0x2e/0x50 [ 7528.156064] kvm_wait+0xcd/0xe0 [ 7528.156991] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7528.158577] do_raw_spin_lock+0x177/0x190 [ 7528.160013] _raw_spin_lock+0x21/0x30 [ 7528.161166] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7528.163167] ? do_raw_spin_unlock+0x75/0x190 [ 7528.164349] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7528.166351] ? do_wait_intr+0xf0/0xf0 [ 7528.167448] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7528.169194] mdc_close+0x53b/0xd70 [mdc] [ 7528.170913] lmv_close+0x18d/0x490 [lmv] [ 7528.175746] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7528.177472] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7528.186795] ll_md_close+0x274/0x800 [lustre] [ 7528.188277] ll_file_release+0x128/0xc50 [lustre] [ 7528.192639] __fput+0x10d/0x440 [ 7528.194873] ____fput+0x12/0x20 [ 7528.196905] task_work_run+0xc6/0x110 [ 7528.203252] exit_to_usermode_loop+0x1dd/0x1f0 [ 7528.207752] do_syscall_64+0x3d6/0x440 [ 7528.210009] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7528.212881] RIP: 0033:0x7fdc0afbfcd8 [ 7528.214992] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7528.225170] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7528.228098] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7528.230145] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7528.233992] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7528.237271] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7528.239978] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7540.053370] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 7540.058047] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7540.095706] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7540.100949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7540.109821] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7540.115574] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7540.134038] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7540.138359] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7540.145916] RDX: 0000000000003d2b RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7540.153510] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7540.157014] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7540.164595] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7540.174235] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7540.182876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7540.187191] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7540.196328] Call Trace: [ 7540.198814] [ 7540.199421] ? show_regs.cold.9+0x22/0x2f [ 7540.201591] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7540.202864] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7540.208128] ? watchdog+0x40/0x40 [ 7540.209628] ? hrtimer_interrupt+0x128/0x2c0 [ 7540.211590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7540.216202] ? apic_timer_interrupt+0xf/0x20 [ 7540.218470] [ 7540.219109] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7540.220730] do_raw_spin_lock+0x177/0x190 [ 7540.221777] _raw_spin_lock+0x21/0x30 [ 7540.222492] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7540.223864] ? do_raw_spin_unlock+0x75/0x190 [ 7540.227716] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7540.230802] ? do_wait_intr+0xf0/0xf0 [ 7540.232040] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7540.240441] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7540.243184] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7540.246660] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7540.248675] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7540.250762] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7540.252490] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7540.253895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7540.255665] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7540.258782] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7540.262679] lmv_intent_open+0x338/0x1150 [lmv] [ 7540.264261] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7540.266891] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7540.268963] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7540.270879] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7540.273455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7540.276964] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7540.281287] ? do_raw_spin_unlock+0x75/0x190 [ 7540.283671] ? _raw_spin_unlock+0x12/0x30 [ 7540.285056] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7540.287460] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7540.290411] ll_atomic_open+0x36a/0xd70 [lustre] [ 7540.293221] lookup_open+0x356/0xb40 [ 7540.295229] do_last+0x313/0xfc0 [ 7540.296883] ? nd_jump_root+0xe5/0x160 [ 7540.301609] ? path_init+0x437/0x520 [ 7540.303678] path_openat+0xf7/0x500 [ 7540.305460] ? do_raw_spin_unlock+0x75/0x190 [ 7540.307657] do_filp_open+0x99/0x140 [ 7540.308980] ? getname_flags+0x6e/0x330 [ 7540.311536] ? __check_object_size+0xff/0x256 [ 7540.313622] ? do_raw_spin_unlock+0x75/0x190 [ 7540.315785] ? _raw_spin_unlock+0x12/0x30 [ 7540.317711] do_sys_openat2+0x2b4/0x410 [ 7540.319674] do_sys_open+0x73/0xa0 [ 7540.320769] __x64_sys_openat+0x24/0x30 [ 7540.325602] do_syscall_64+0xc1/0x440 [ 7540.326679] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7540.328678] RIP: 0033:0x7fdc0afbf332 [ 7540.329967] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7540.336552] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7540.342101] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7540.346238] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7540.348735] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7540.351791] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7540.356606] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7544.048034] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7544.051040] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7544.081613] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7544.090868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7544.096496] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7544.099276] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7544.109636] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7544.114045] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7544.118596] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7544.123503] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7544.126793] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7544.136049] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7544.145701] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7544.150088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7544.154879] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7544.163830] Call Trace: [ 7544.165536] [ 7544.166473] ? show_regs.cold.9+0x22/0x2f [ 7544.172137] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7544.175147] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7544.177479] ? watchdog+0x40/0x40 [ 7544.179109] ? hrtimer_interrupt+0x128/0x2c0 [ 7544.181207] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7544.184404] ? apic_timer_interrupt+0xf/0x20 [ 7544.186088] [ 7544.187108] ? native_safe_halt+0x2e/0x50 [ 7544.188866] kvm_wait+0xcd/0xe0 [ 7544.190500] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7544.192811] ? dump_stack+0xf6/0x10e [ 7544.194291] do_raw_spin_lock+0x177/0x190 [ 7544.195849] _raw_spin_lock+0x21/0x30 [ 7544.197430] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7544.199809] obd_mod_free+0x46/0x140 [ptlrpc] [ 7544.201672] mdc_commit_open+0x91/0xb0 [mdc] [ 7544.203675] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7544.205270] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7544.207651] after_reply+0xe2c/0x1780 [ptlrpc] [ 7544.210073] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7544.212536] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7544.214758] ? do_wait_intr+0xf0/0xf0 [ 7544.216177] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7544.218670] kthread+0x1d1/0x200 [ 7544.219948] ? set_kthread_struct+0x70/0x70 [ 7544.221861] ret_from_fork+0x1f/0x30 [ 7548.051034] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7548.054649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7548.101402] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7548.108170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7548.113654] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7548.122769] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7548.128310] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7548.146733] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7548.151919] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7548.159998] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7548.162471] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7548.168645] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7548.172663] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7548.179703] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7548.189267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7548.191661] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7548.197246] Call Trace: [ 7548.198473] [ 7548.199594] ? show_regs.cold.9+0x22/0x2f [ 7548.204689] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7548.208826] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7548.211680] ? watchdog+0x40/0x40 [ 7548.212913] ? hrtimer_interrupt+0x128/0x2c0 [ 7548.218666] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7548.221571] ? apic_timer_interrupt+0xf/0x20 [ 7548.224782] [ 7548.225712] ? native_safe_halt+0x2e/0x50 [ 7548.227780] kvm_wait+0xcd/0xe0 [ 7548.230075] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7548.239690] do_raw_spin_lock+0x177/0x190 [ 7548.242287] _raw_spin_lock+0x21/0x30 [ 7548.243592] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7548.245584] process_one_work+0x2c8/0x700 [ 7548.253803] worker_thread+0x296/0x6e0 [ 7548.258661] ? rescuer_thread+0x570/0x570 [ 7548.260238] kthread+0x1d1/0x200 [ 7548.261451] ? set_kthread_struct+0x70/0x70 [ 7548.269241] ret_from_fork+0x1f/0x30 [ 7548.896194] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5523s! [ 7548.913948] Showing busy workqueues and worker pools: [ 7548.924173] workqueue events: flags=0x0 [ 7548.938712] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7548.945590] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7548.955175] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7548.958304] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7548.963105] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7548.966060] pending: kfree_rcu_monitor [ 7548.967471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7548.969709] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7548.975096] workqueue events_long: flags=0x0 [ 7548.977090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7548.979512] pending: lu_objects_flush [obdclass] [ 7548.981423] workqueue events_unbound: flags=0x2 [ 7548.983367] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7548.986472] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7548.990759] workqueue events_power_efficient: flags=0x80 [ 7548.992753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7548.995056] pending: neigh_periodic_work, check_lifetime [ 7548.997514] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7548.999621] pending: do_cache_clean, neigh_periodic_work [ 7549.002143] workqueue mm_percpu_wq: flags=0x8 [ 7549.003662] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7549.006295] pending: lru_add_drain_per_cpu [ 7549.007536] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7549.010373] pending: vmstat_update, lru_add_drain_per_cpu [ 7549.012423] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7549.014663] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7549.017044] workqueue writeback: flags=0x4a [ 7549.018224] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7549.020133] pending: wb_workfn, wb_workfn [ 7549.021409] workqueue xprtiod: flags=0xa [ 7549.022562] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7549.024684] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7549.027292] workqueue ipv6_addrconf: flags=0x40008 [ 7549.028731] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7549.030555] pending: addrconf_verify_work [ 7549.032136] workqueue knbd0-recv: flags=0x1a [ 7549.033417] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7549.035668] in-flight: 655:recv_work [ 7549.037203] workqueue ptlrpc_pinger: flags=0x2 [ 7549.038804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7549.041410] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7549.043681] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5524s workers=4 idle: 36396 38 11188 [ 7549.046717] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7539s workers=2 idle: 658 [ 7556.054027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 7556.062011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7556.091797] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7556.098029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7556.107248] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7556.109579] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7556.130821] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7556.144464] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7556.150147] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7556.155881] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7556.160709] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7556.164804] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7556.167421] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7556.172612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7556.174891] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7556.180871] Call Trace: [ 7556.181951] [ 7556.182964] ? show_regs.cold.9+0x22/0x2f [ 7556.184680] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7556.185901] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7556.187555] ? watchdog+0x40/0x40 [ 7556.188724] ? hrtimer_interrupt+0x128/0x2c0 [ 7556.190024] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7556.191730] ? apic_timer_interrupt+0xf/0x20 [ 7556.192958] [ 7556.193759] ? native_safe_halt+0x2e/0x50 [ 7556.194947] kvm_wait+0xcd/0xe0 [ 7556.195838] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7556.197729] do_raw_spin_lock+0x177/0x190 [ 7556.198850] _raw_spin_lock+0x21/0x30 [ 7556.199910] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7556.201886] ? do_raw_spin_unlock+0x75/0x190 [ 7556.203170] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7556.206847] ? do_wait_intr+0xf0/0xf0 [ 7556.208410] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7556.210926] mdc_close+0x53b/0xd70 [mdc] [ 7556.212863] lmv_close+0x18d/0x490 [lmv] [ 7556.213986] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7556.218176] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7556.222071] ll_md_close+0x274/0x800 [lustre] [ 7556.224548] ll_file_release+0x128/0xc50 [lustre] [ 7556.227640] __fput+0x10d/0x440 [ 7556.230722] ____fput+0x12/0x20 [ 7556.231748] task_work_run+0xc6/0x110 [ 7556.233640] exit_to_usermode_loop+0x1dd/0x1f0 [ 7556.237838] do_syscall_64+0x3d6/0x440 [ 7556.239824] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7556.241828] RIP: 0033:0x7fdc0afbfcd8 [ 7556.244792] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7556.253614] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7556.257046] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7556.268307] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7556.274593] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7556.277796] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7556.281613] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7568.052067] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 7568.056681] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7568.098620] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7568.108480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7568.112063] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7568.114902] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7568.123731] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7568.126973] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7568.129510] RDX: 000000000000682f RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7568.131837] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7568.134430] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7568.137082] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7568.140244] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7568.143539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7568.150444] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7568.158600] Call Trace: [ 7568.161051] [ 7568.161665] ? show_regs.cold.9+0x22/0x2f [ 7568.163095] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7568.167226] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7568.171203] ? watchdog+0x40/0x40 [ 7568.173666] ? hrtimer_interrupt+0x128/0x2c0 [ 7568.176896] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7568.182585] ? apic_timer_interrupt+0xf/0x20 [ 7568.188737] [ 7568.192468] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7568.201422] do_raw_spin_lock+0x177/0x190 [ 7568.203656] _raw_spin_lock+0x21/0x30 [ 7568.205865] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7568.208889] ? do_raw_spin_unlock+0x75/0x190 [ 7568.210862] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7568.213870] ? do_wait_intr+0xf0/0xf0 [ 7568.215157] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7568.224637] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7568.229201] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7568.234006] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7568.238156] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7568.245843] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7568.250214] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7568.262094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7568.270039] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7568.275722] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7568.279637] lmv_intent_open+0x338/0x1150 [lmv] [ 7568.289979] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7568.296677] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7568.300901] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7568.306293] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7568.311046] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7568.316158] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7568.319615] ? do_raw_spin_unlock+0x75/0x190 [ 7568.322237] ? _raw_spin_unlock+0x12/0x30 [ 7568.332061] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7568.335944] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7568.340278] ll_atomic_open+0x36a/0xd70 [lustre] [ 7568.342969] lookup_open+0x356/0xb40 [ 7568.344382] do_last+0x313/0xfc0 [ 7568.350700] ? nd_jump_root+0xe5/0x160 [ 7568.357229] ? path_init+0x437/0x520 [ 7568.368621] path_openat+0xf7/0x500 [ 7568.370813] ? do_raw_spin_unlock+0x75/0x190 [ 7568.378020] do_filp_open+0x99/0x140 [ 7568.380306] ? getname_flags+0x6e/0x330 [ 7568.383941] ? __check_object_size+0xff/0x256 [ 7568.386887] ? do_raw_spin_unlock+0x75/0x190 [ 7568.392433] ? _raw_spin_unlock+0x12/0x30 [ 7568.396786] do_sys_openat2+0x2b4/0x410 [ 7568.400142] do_sys_open+0x73/0xa0 [ 7568.402340] __x64_sys_openat+0x24/0x30 [ 7568.404134] do_syscall_64+0xc1/0x440 [ 7568.406935] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7568.410981] RIP: 0033:0x7fdc0afbf332 [ 7568.412707] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7568.427646] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7568.431614] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7568.435752] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7568.444144] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7568.449244] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7568.459722] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7576.048105] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7576.052291] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7576.056821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7576.057259] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7576.057267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7576.057270] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7576.057285] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7576.057289] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7576.057293] RAX: 0000000000000278 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7576.057296] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7576.057298] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7576.057300] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7576.057302] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7576.057306] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7576.057309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7576.057312] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7576.057321] Call Trace: [ 7576.057326] [ 7576.057334] ? show_regs.cold.9+0x22/0x2f [ 7576.057344] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7576.063586] Modules linked in: [ 7576.078889] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7576.078909] ? watchdog+0x40/0x40 [ 7576.078915] ? hrtimer_interrupt+0x128/0x2c0 [ 7576.078919] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7576.078926] ? apic_timer_interrupt+0xf/0x20 [ 7576.078931] [ 7576.078936] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7576.078943] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7576.078947] ? dump_stack+0xf6/0x10e [ 7576.078953] do_raw_spin_lock+0x177/0x190 [ 7576.078959] _raw_spin_lock+0x21/0x30 [ 7576.078967] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7576.081341] lustre(O) [ 7576.083044] obd_mod_free+0x46/0x140 [ptlrpc] [ 7576.084334] osp(O) [ 7576.089585] mdc_commit_open+0x91/0xb0 [mdc] [ 7576.089632] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7576.091018] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7576.091091] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7576.091096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7576.091103] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7576.091360] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7576.091370] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7576.091374] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7576.091378] RAX: 0000000000007e98 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7576.091381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7576.091383] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7576.091385] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7576.091388] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7576.091391] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7576.091394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7576.091396] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7576.091405] Call Trace: [ 7576.091409] [ 7576.091412] ? show_regs.cold.9+0x22/0x2f [ 7576.091420] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7576.091426] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7576.091430] ? watchdog+0x40/0x40 [ 7576.091434] ? hrtimer_interrupt+0x128/0x2c0 [ 7576.091439] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7576.091448] ? apic_timer_interrupt+0xf/0x20 [ 7576.091452] [ 7576.091455] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7576.091459] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7576.091465] do_raw_spin_lock+0x177/0x190 [ 7576.091469] _raw_spin_lock+0x21/0x30 [ 7576.091474] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7576.091738] process_one_work+0x2c8/0x700 [ 7576.091745] worker_thread+0x296/0x6e0 [ 7576.091749] ? rescuer_thread+0x570/0x570 [ 7576.091752] kthread+0x1d1/0x200 [ 7576.091757] ? set_kthread_struct+0x70/0x70 [ 7576.091761] ret_from_fork+0x1f/0x30 [ 7576.359331] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7576.365793] after_reply+0xe2c/0x1780 [ptlrpc] [ 7576.373070] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7576.381608] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7576.389492] ? do_wait_intr+0xf0/0xf0 [ 7576.391288] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7576.395309] kthread+0x1d1/0x200 [ 7576.397060] ? set_kthread_struct+0x70/0x70 [ 7576.399592] ret_from_fork+0x1f/0x30 [ 7579.104840] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5554s! [ 7579.115098] Showing busy workqueues and worker pools: [ 7579.120386] workqueue events: flags=0x0 [ 7579.128136] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7579.138968] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7579.145299] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7579.151074] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7579.158377] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7579.164960] pending: kfree_rcu_monitor [ 7579.168787] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7579.173477] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7579.179088] workqueue events_long: flags=0x0 [ 7579.182274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7579.186034] pending: lu_objects_flush [obdclass] [ 7579.187859] workqueue events_unbound: flags=0x2 [ 7579.193064] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7579.198084] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7579.207874] workqueue events_power_efficient: flags=0x80 [ 7579.210627] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7579.214096] pending: neigh_periodic_work, check_lifetime [ 7579.217033] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7579.218810] pending: do_cache_clean, neigh_periodic_work [ 7579.226273] workqueue mm_percpu_wq: flags=0x8 [ 7579.228875] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7579.232154] pending: lru_add_drain_per_cpu [ 7579.234690] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7579.240557] pending: vmstat_update, lru_add_drain_per_cpu [ 7579.246010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7579.249869] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7579.256853] workqueue writeback: flags=0x4a [ 7579.259745] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7579.262050] pending: wb_workfn, wb_workfn [ 7579.264034] workqueue xprtiod: flags=0xa [ 7579.266682] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7579.270042] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7579.279840] workqueue ipv6_addrconf: flags=0x40008 [ 7579.281295] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7579.282902] pending: addrconf_verify_work [ 7579.286262] workqueue knbd0-recv: flags=0x1a [ 7579.292815] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7579.297472] in-flight: 655:recv_work [ 7579.299158] workqueue ptlrpc_pinger: flags=0x2 [ 7579.301065] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7579.303903] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7579.308084] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5554s workers=4 idle: 36396 38 11188 [ 7579.311528] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7569s workers=2 idle: 658 [ 7584.054057] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 7584.061552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7584.085330] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7584.091172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7584.094960] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7584.098964] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7584.113580] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7584.117485] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7584.121301] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7584.126490] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7584.132650] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7584.138754] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7584.147245] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7584.153099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7584.155782] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7584.159695] Call Trace: [ 7584.160701] [ 7584.161548] ? show_regs.cold.9+0x22/0x2f [ 7584.163036] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7584.165625] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7584.168153] ? watchdog+0x40/0x40 [ 7584.169227] ? hrtimer_interrupt+0x128/0x2c0 [ 7584.170631] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7584.172313] ? apic_timer_interrupt+0xf/0x20 [ 7584.174898] [ 7584.175504] ? native_safe_halt+0x2e/0x50 [ 7584.178326] kvm_wait+0xcd/0xe0 [ 7584.179317] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7584.181678] do_raw_spin_lock+0x177/0x190 [ 7584.184009] _raw_spin_lock+0x21/0x30 [ 7584.191949] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7584.194000] ? do_raw_spin_unlock+0x75/0x190 [ 7584.196015] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7584.198947] ? do_wait_intr+0xf0/0xf0 [ 7584.201712] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7584.204517] mdc_close+0x53b/0xd70 [mdc] [ 7584.207100] lmv_close+0x18d/0x490 [lmv] [ 7584.209244] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7584.214333] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7584.215960] ll_md_close+0x274/0x800 [lustre] [ 7584.219259] ll_file_release+0x128/0xc50 [lustre] [ 7584.221859] __fput+0x10d/0x440 [ 7584.223099] ____fput+0x12/0x20 [ 7584.224737] task_work_run+0xc6/0x110 [ 7584.226276] exit_to_usermode_loop+0x1dd/0x1f0 [ 7584.228159] do_syscall_64+0x3d6/0x440 [ 7584.229836] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7584.232320] RIP: 0033:0x7fdc0afbfcd8 [ 7584.233977] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7584.248357] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7584.253036] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7584.262105] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7584.266911] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7584.271128] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7584.275649] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7596.052028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7596.057076] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7596.115779] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7596.121623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7596.132290] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7596.136532] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7596.146783] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7596.150182] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7596.152711] RDX: 0000000000007e96 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7596.155852] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7596.158735] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7596.161700] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7596.164921] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7596.167368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7596.169688] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7596.172824] Call Trace: [ 7596.173800] [ 7596.174743] ? show_regs.cold.9+0x22/0x2f [ 7596.176223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7596.178218] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7596.179732] ? watchdog+0x40/0x40 [ 7596.181029] ? hrtimer_interrupt+0x128/0x2c0 [ 7596.182384] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7596.184379] ? apic_timer_interrupt+0xf/0x20 [ 7596.186152] [ 7596.186912] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7596.188911] do_raw_spin_lock+0x177/0x190 [ 7596.190895] _raw_spin_lock+0x21/0x30 [ 7596.192398] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7596.195099] ? do_raw_spin_unlock+0x75/0x190 [ 7596.196801] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7596.199574] ? do_wait_intr+0xf0/0xf0 [ 7596.200990] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7596.203291] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7596.206157] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7596.208707] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7596.210991] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7596.213211] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7596.215175] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7596.216879] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7596.218970] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7596.221418] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7596.223271] lmv_intent_open+0x338/0x1150 [lmv] [ 7596.225381] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7596.227146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7596.229085] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7596.230922] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7596.232886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7596.235581] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7596.237725] ? do_raw_spin_unlock+0x75/0x190 [ 7596.239711] ? _raw_spin_unlock+0x12/0x30 [ 7596.241622] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7596.244684] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7596.247803] ll_atomic_open+0x36a/0xd70 [lustre] [ 7596.250348] lookup_open+0x356/0xb40 [ 7596.253007] do_last+0x313/0xfc0 [ 7596.255027] ? nd_jump_root+0xe5/0x160 [ 7596.256652] ? path_init+0x437/0x520 [ 7596.259582] path_openat+0xf7/0x500 [ 7596.261153] ? do_raw_spin_unlock+0x75/0x190 [ 7596.264152] do_filp_open+0x99/0x140 [ 7596.265755] ? getname_flags+0x6e/0x330 [ 7596.267729] ? __check_object_size+0xff/0x256 [ 7596.270336] ? do_raw_spin_unlock+0x75/0x190 [ 7596.274677] ? _raw_spin_unlock+0x12/0x30 [ 7596.276763] do_sys_openat2+0x2b4/0x410 [ 7596.280609] do_sys_open+0x73/0xa0 [ 7596.283754] __x64_sys_openat+0x24/0x30 [ 7596.286144] do_syscall_64+0xc1/0x440 [ 7596.289308] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7596.296354] RIP: 0033:0x7fdc0afbf332 [ 7596.297565] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7596.311929] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7596.314562] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7596.317885] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7596.325958] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7596.328947] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7596.334140] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7604.048106] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7604.050559] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7604.051682] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7604.078089] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7604.078102] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7604.078105] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7604.078118] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7604.078122] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7604.078127] RAX: 0000000000005260 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7604.078130] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7604.078132] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7604.078134] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7604.078145] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7604.078149] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7604.078151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7604.078154] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7604.078161] Call Trace: [ 7604.078165] [ 7604.078171] ? show_regs.cold.9+0x22/0x2f [ 7604.078178] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7604.078184] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7604.078188] ? watchdog+0x40/0x40 [ 7604.078192] ? hrtimer_interrupt+0x128/0x2c0 [ 7604.078196] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7604.078203] ? apic_timer_interrupt+0xf/0x20 [ 7604.078207] [ 7604.078210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7604.078214] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7604.078218] ? dump_stack+0xf6/0x10e [ 7604.078223] do_raw_spin_lock+0x177/0x190 [ 7604.078227] _raw_spin_lock+0x21/0x30 [ 7604.078235] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7604.078755] obd_mod_free+0x46/0x140 [ptlrpc] [ 7604.079008] mdc_commit_open+0x91/0xb0 [mdc] [ 7604.079043] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7604.079288] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7604.079618] after_reply+0xe2c/0x1780 [ptlrpc] [ 7604.080210] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7604.080427] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7604.080627] ? do_wait_intr+0xf0/0xf0 [ 7604.080635] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7604.080795] kthread+0x1d1/0x200 [ 7604.080800] ? set_kthread_struct+0x70/0x70 [ 7604.080804] ret_from_fork+0x1f/0x30 [ 7604.288536] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7604.321530] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7604.325787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7604.328866] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7604.334891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7604.338205] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7604.346038] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7604.349791] RAX: 0000000000005b3c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7604.352517] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7604.355896] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7604.359263] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7604.362489] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7604.366029] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7604.369280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7604.374163] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7604.382883] Call Trace: [ 7604.385868] [ 7604.387070] ? show_regs.cold.9+0x22/0x2f [ 7604.389115] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7604.393840] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7604.397518] ? watchdog+0x40/0x40 [ 7604.399198] ? hrtimer_interrupt+0x128/0x2c0 [ 7604.401132] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7604.403785] ? apic_timer_interrupt+0xf/0x20 [ 7604.405293] [ 7604.406376] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7604.409237] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7604.412458] do_raw_spin_lock+0x177/0x190 [ 7604.415573] _raw_spin_lock+0x21/0x30 [ 7604.418433] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7604.420926] process_one_work+0x2c8/0x700 [ 7604.422788] worker_thread+0x296/0x6e0 [ 7604.424276] ? rescuer_thread+0x570/0x570 [ 7604.426211] kthread+0x1d1/0x200 [ 7604.428498] ? set_kthread_struct+0x70/0x70 [ 7604.431001] ret_from_fork+0x1f/0x30 [ 7609.826398] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5584s! [ 7609.837416] Showing busy workqueues and worker pools: [ 7609.847652] workqueue events: flags=0x0 [ 7609.853811] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7609.864676] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7609.875221] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7609.882612] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7609.899375] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7609.909099] pending: kfree_rcu_monitor [ 7609.911715] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7609.916068] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7609.927053] workqueue events_long: flags=0x0 [ 7609.930398] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7609.938359] pending: lu_objects_flush [obdclass] [ 7609.944837] workqueue events_unbound: flags=0x2 [ 7609.950826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7609.960144] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7609.971715] workqueue events_power_efficient: flags=0x80 [ 7609.977817] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7609.985432] pending: neigh_periodic_work, check_lifetime [ 7609.990043] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7609.998953] pending: do_cache_clean, neigh_periodic_work [ 7610.004723] workqueue mm_percpu_wq: flags=0x8 [ 7610.011612] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7610.016737] pending: lru_add_drain_per_cpu [ 7610.021108] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7610.028999] pending: vmstat_update, lru_add_drain_per_cpu [ 7610.034184] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7610.048141] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7610.053406] workqueue writeback: flags=0x4a [ 7610.058307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7610.063884] pending: wb_workfn, wb_workfn [ 7610.066189] workqueue xprtiod: flags=0xa [ 7610.075822] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7610.086874] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7610.101362] workqueue ipv6_addrconf: flags=0x40008 [ 7610.105415] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7610.108357] pending: addrconf_verify_work [ 7610.110567] workqueue knbd0-recv: flags=0x1a [ 7610.112705] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7610.117802] in-flight: 655:recv_work [ 7610.119344] workqueue ptlrpc_pinger: flags=0x2 [ 7610.121798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7610.124062] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7610.136387] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5585s workers=4 idle: 36396 38 11188 [ 7610.141789] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7600s workers=2 idle: 658 [ 7612.054675] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 7612.060689] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7612.080640] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7612.084103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7612.087288] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7612.089785] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7612.101105] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7612.112547] RAX: 0000000000007abb RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7612.124887] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7612.141297] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7612.152546] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7612.161507] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7612.167935] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7612.170047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7612.171596] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7612.173680] Call Trace: [ 7612.174551] [ 7612.175084] ? show_regs.cold.9+0x22/0x2f [ 7612.176325] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7612.177776] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7612.179069] ? watchdog+0x40/0x40 [ 7612.180414] ? hrtimer_interrupt+0x128/0x2c0 [ 7612.181867] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7612.183390] ? apic_timer_interrupt+0xf/0x20 [ 7612.184697] [ 7612.185489] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7612.187016] do_raw_spin_lock+0x177/0x190 [ 7612.188086] _raw_spin_lock+0x21/0x30 [ 7612.189401] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7612.191339] ? do_raw_spin_unlock+0x75/0x190 [ 7612.192553] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7612.194474] ? do_wait_intr+0xf0/0xf0 [ 7612.195567] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7612.197236] mdc_close+0x53b/0xd70 [mdc] [ 7612.198342] lmv_close+0x18d/0x490 [lmv] [ 7612.199330] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7612.200848] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7612.202158] ll_md_close+0x274/0x800 [lustre] [ 7612.203401] ll_file_release+0x128/0xc50 [lustre] [ 7612.204746] __fput+0x10d/0x440 [ 7612.205543] ____fput+0x12/0x20 [ 7612.206318] task_work_run+0xc6/0x110 [ 7612.207168] exit_to_usermode_loop+0x1dd/0x1f0 [ 7612.208606] do_syscall_64+0x3d6/0x440 [ 7612.209692] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7612.211059] RIP: 0033:0x7fdc0afbfcd8 [ 7612.212084] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7612.216908] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7612.218865] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7612.220700] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7612.222556] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7612.224472] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7612.226324] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7624.052045] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7624.054470] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7624.090796] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7624.098209] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7624.100818] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7624.101922] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7624.118596] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7624.124496] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7624.129764] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7624.132782] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7624.135733] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7624.138012] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7624.140892] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7624.147468] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7624.149850] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7624.152508] Call Trace: [ 7624.153367] [ 7624.154901] ? show_regs.cold.9+0x22/0x2f [ 7624.157473] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7624.159827] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7624.162223] ? watchdog+0x40/0x40 [ 7624.163674] ? hrtimer_interrupt+0x128/0x2c0 [ 7624.165159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7624.166925] ? apic_timer_interrupt+0xf/0x20 [ 7624.168501] [ 7624.169124] ? native_safe_halt+0x2e/0x50 [ 7624.170477] kvm_wait+0xcd/0xe0 [ 7624.171795] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7624.173246] do_raw_spin_lock+0x177/0x190 [ 7624.178123] _raw_spin_lock+0x21/0x30 [ 7624.179510] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7624.181559] ? do_raw_spin_unlock+0x75/0x190 [ 7624.184818] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7624.186711] ? do_wait_intr+0xf0/0xf0 [ 7624.188536] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7624.191453] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7624.195664] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7624.198634] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7624.200926] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7624.202581] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7624.204456] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7624.205934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7624.207827] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7624.209918] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7624.211927] lmv_intent_open+0x338/0x1150 [lmv] [ 7624.215715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7624.217591] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7624.219514] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7624.221078] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7624.223395] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7624.225339] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7624.227332] ? do_raw_spin_unlock+0x75/0x190 [ 7624.228586] ? _raw_spin_unlock+0x12/0x30 [ 7624.233842] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7624.235905] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7624.237656] ll_atomic_open+0x36a/0xd70 [lustre] [ 7624.239607] lookup_open+0x356/0xb40 [ 7624.240850] do_last+0x313/0xfc0 [ 7624.245949] ? nd_jump_root+0xe5/0x160 [ 7624.248946] ? path_init+0x437/0x520 [ 7624.250882] path_openat+0xf7/0x500 [ 7624.253259] ? do_raw_spin_unlock+0x75/0x190 [ 7624.254227] do_filp_open+0x99/0x140 [ 7624.255011] ? getname_flags+0x6e/0x330 [ 7624.255816] ? __check_object_size+0xff/0x256 [ 7624.257654] ? do_raw_spin_unlock+0x75/0x190 [ 7624.258929] ? _raw_spin_unlock+0x12/0x30 [ 7624.260473] do_sys_openat2+0x2b4/0x410 [ 7624.262326] do_sys_open+0x73/0xa0 [ 7624.263822] __x64_sys_openat+0x24/0x30 [ 7624.264917] do_syscall_64+0xc1/0x440 [ 7624.267328] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7624.270757] RIP: 0033:0x7fdc0afbf332 [ 7624.272850] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7624.284762] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7624.287068] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7624.288926] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7624.294057] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7624.298428] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7624.304466] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7632.052788] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7632.056810] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7632.094873] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7632.107655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7632.114012] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7632.120070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7632.123251] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7632.136775] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7632.140634] RAX: 0000000000007d84 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7632.143728] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7632.150759] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7632.156615] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7632.161165] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7632.164625] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7632.168433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7632.172183] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7632.183529] Call Trace: [ 7632.184719] [ 7632.187622] ? show_regs.cold.9+0x22/0x2f [ 7632.191790] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7632.198600] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7632.203605] ? watchdog+0x40/0x40 [ 7632.204896] ? hrtimer_interrupt+0x128/0x2c0 [ 7632.207911] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7632.209701] ? apic_timer_interrupt+0xf/0x20 [ 7632.213356] [ 7632.214542] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7632.220892] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7632.222801] do_raw_spin_lock+0x177/0x190 [ 7632.226230] _raw_spin_lock+0x21/0x30 [ 7632.228805] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7632.234185] process_one_work+0x2c8/0x700 [ 7632.235797] worker_thread+0x296/0x6e0 [ 7632.239505] ? rescuer_thread+0x570/0x570 [ 7632.242820] kthread+0x1d1/0x200 [ 7632.245449] ? set_kthread_struct+0x70/0x70 [ 7632.250665] ret_from_fork+0x1f/0x30 [ 7636.048128] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7636.053968] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7636.081225] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7636.087017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7636.091490] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7636.093959] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7636.109544] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7636.114018] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7636.118555] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7636.123222] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7636.126983] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7636.130694] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7636.136228] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7636.139794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7636.142774] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7636.145082] Call Trace: [ 7636.145948] [ 7636.147209] ? show_regs.cold.9+0x22/0x2f [ 7636.149510] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7636.152533] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7636.155523] ? watchdog+0x40/0x40 [ 7636.157454] ? hrtimer_interrupt+0x128/0x2c0 [ 7636.159753] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7636.161739] ? apic_timer_interrupt+0xf/0x20 [ 7636.164145] [ 7636.165049] ? native_safe_halt+0x2e/0x50 [ 7636.167342] kvm_wait+0xcd/0xe0 [ 7636.168746] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7636.171604] ? dump_stack+0xf6/0x10e [ 7636.172964] do_raw_spin_lock+0x177/0x190 [ 7636.175706] _raw_spin_lock+0x21/0x30 [ 7636.177572] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7636.180540] obd_mod_free+0x46/0x140 [ptlrpc] [ 7636.183085] mdc_commit_open+0x91/0xb0 [mdc] [ 7636.185269] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7636.188478] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7636.191145] after_reply+0xe2c/0x1780 [ptlrpc] [ 7636.193660] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7636.196972] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7636.199572] ? do_wait_intr+0xf0/0xf0 [ 7636.201242] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7636.203993] kthread+0x1d1/0x200 [ 7636.205945] ? set_kthread_struct+0x70/0x70 [ 7636.207824] ret_from_fork+0x1f/0x30 [ 7640.055767] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7640.059026] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7640.087411] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7640.092001] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7640.095470] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7640.097618] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7640.111570] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7640.115110] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7640.118280] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7640.121509] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7640.123918] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7640.134378] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7640.141765] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7640.151612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7640.157707] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7640.161885] Call Trace: [ 7640.163078] [ 7640.164817] ? show_regs.cold.9+0x22/0x2f [ 7640.175555] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7640.183415] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7640.189825] ? watchdog+0x40/0x40 [ 7640.197190] ? hrtimer_interrupt+0x128/0x2c0 [ 7640.205518] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7640.213602] ? apic_timer_interrupt+0xf/0x20 [ 7640.215397] [ 7640.216256] ? native_safe_halt+0x2e/0x50 [ 7640.220106] kvm_wait+0xcd/0xe0 [ 7640.223399] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7640.228425] do_raw_spin_lock+0x177/0x190 [ 7640.231332] _raw_spin_lock+0x21/0x30 [ 7640.236608] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7640.239520] ? do_raw_spin_unlock+0x75/0x190 [ 7640.242559] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7640.246339] ? do_wait_intr+0xf0/0xf0 [ 7640.247960] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7640.250594] mdc_close+0x53b/0xd70 [mdc] [ 7640.252290] lmv_close+0x18d/0x490 [lmv] [ 7640.254117] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7640.256082] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7640.258657] ll_md_close+0x274/0x800 [lustre] [ 7640.261299] ll_file_release+0x128/0xc50 [lustre] [ 7640.263905] __fput+0x10d/0x440 [ 7640.265739] ____fput+0x12/0x20 [ 7640.267623] task_work_run+0xc6/0x110 [ 7640.270657] exit_to_usermode_loop+0x1dd/0x1f0 [ 7640.275013] do_syscall_64+0x3d6/0x440 [ 7640.276386] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7640.280902] RIP: 0033:0x7fdc0afbfcd8 [ 7640.282536] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7640.290994] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7640.294520] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7640.298683] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7640.304801] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7640.308213] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7640.313650] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7640.543046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5615s! [ 7640.546040] Showing busy workqueues and worker pools: [ 7640.547918] workqueue events: flags=0x0 [ 7640.553957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7640.564037] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7640.574197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7640.580604] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7640.592337] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7640.604192] pending: kfree_rcu_monitor [ 7640.605563] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7640.614410] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7640.623976] workqueue events_long: flags=0x0 [ 7640.625232] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7640.628226] pending: lu_objects_flush [obdclass] [ 7640.633353] workqueue events_unbound: flags=0x2 [ 7640.635418] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7640.639420] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7640.647050] workqueue events_power_efficient: flags=0x80 [ 7640.651046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7640.656043] pending: neigh_periodic_work, check_lifetime [ 7640.659297] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7640.663316] pending: do_cache_clean, neigh_periodic_work [ 7640.669138] workqueue mm_percpu_wq: flags=0x8 [ 7640.671552] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7640.673976] pending: lru_add_drain_per_cpu [ 7640.675749] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7640.678271] pending: vmstat_update, lru_add_drain_per_cpu [ 7640.684269] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7640.687390] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7640.689871] workqueue writeback: flags=0x4a [ 7640.692393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7640.696256] pending: wb_workfn, wb_workfn [ 7640.698377] workqueue xprtiod: flags=0xa [ 7640.700285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7640.703715] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7640.707676] workqueue ipv6_addrconf: flags=0x40008 [ 7640.711324] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7640.715470] pending: addrconf_verify_work [ 7640.718127] workqueue knbd0-recv: flags=0x1a [ 7640.720300] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7640.724393] in-flight: 655:recv_work [ 7640.726912] workqueue ptlrpc_pinger: flags=0x2 [ 7640.730100] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7640.734019] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7640.737386] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5615s workers=4 idle: 36396 38 11188 [ 7640.743181] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7631s workers=2 idle: 658 [ 7652.052028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7652.057204] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7652.085981] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7652.093044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7652.098301] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7652.101705] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7652.118088] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7652.124864] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7652.130504] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7652.136015] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7652.140924] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7652.146815] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7652.152533] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7652.159530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7652.164316] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7652.169077] Call Trace: [ 7652.170913] [ 7652.172161] ? show_regs.cold.9+0x22/0x2f [ 7652.173428] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7652.175386] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7652.178081] ? watchdog+0x40/0x40 [ 7652.179256] ? hrtimer_interrupt+0x128/0x2c0 [ 7652.181287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7652.183243] ? apic_timer_interrupt+0xf/0x20 [ 7652.185576] [ 7652.186834] ? native_safe_halt+0x2e/0x50 [ 7652.189130] kvm_wait+0xcd/0xe0 [ 7652.190649] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7652.193221] do_raw_spin_lock+0x177/0x190 [ 7652.195220] _raw_spin_lock+0x21/0x30 [ 7652.198319] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7652.201147] ? do_raw_spin_unlock+0x75/0x190 [ 7652.202806] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7652.204857] ? do_wait_intr+0xf0/0xf0 [ 7652.206138] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7652.208946] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7652.211392] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7652.213487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7652.215357] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7652.217431] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7652.219587] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7652.221069] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7652.223385] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7652.225679] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7652.227811] lmv_intent_open+0x338/0x1150 [lmv] [ 7652.229989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7652.233893] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7652.236875] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7652.238941] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7652.240965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7652.242830] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7652.244030] ? do_raw_spin_unlock+0x75/0x190 [ 7652.245749] ? _raw_spin_unlock+0x12/0x30 [ 7652.247050] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7652.249033] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7652.250918] ll_atomic_open+0x36a/0xd70 [lustre] [ 7652.252410] lookup_open+0x356/0xb40 [ 7652.253420] do_last+0x313/0xfc0 [ 7652.254575] ? nd_jump_root+0xe5/0x160 [ 7652.255931] ? path_init+0x437/0x520 [ 7652.257700] path_openat+0xf7/0x500 [ 7652.259138] ? do_raw_spin_unlock+0x75/0x190 [ 7652.260586] do_filp_open+0x99/0x140 [ 7652.261850] ? getname_flags+0x6e/0x330 [ 7652.264111] ? __check_object_size+0xff/0x256 [ 7652.265797] ? do_raw_spin_unlock+0x75/0x190 [ 7652.267349] ? _raw_spin_unlock+0x12/0x30 [ 7652.268932] do_sys_openat2+0x2b4/0x410 [ 7652.270946] do_sys_open+0x73/0xa0 [ 7652.272547] __x64_sys_openat+0x24/0x30 [ 7652.274115] do_syscall_64+0xc1/0x440 [ 7652.275354] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7652.277171] RIP: 0033:0x7fdc0afbf332 [ 7652.278474] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7652.290591] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7652.294047] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7652.301419] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7652.304837] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7652.308903] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7652.312903] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7659.484088] rcu: INFO: rcu_sched self-detected stall on CPU [ 7659.487183] rcu: 0-...!: (3495860 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 7659.493028] (t=5640141 jiffies g=270761 q=4010) [ 7659.495336] rcu: rcu_sched kthread starved for 5635073 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 7659.502775] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7659.507889] rcu: RCU grace-period kthread stack dump: [ 7659.510154] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7659.515895] Call Trace: [ 7659.517883] __schedule+0x351/0xcb0 [ 7659.520195] schedule+0xc0/0x180 [ 7659.522185] schedule_timeout+0xb4/0x190 [ 7659.524554] ? do_raw_spin_unlock+0x75/0x190 [ 7659.526794] ? __next_timer_interrupt+0x160/0x160 [ 7659.529270] ? prepare_to_swait_event+0xf3/0x140 [ 7659.534390] rcu_gp_kthread+0x8dd/0x15b0 [ 7659.536119] ? force_qs_rnp+0x2c0/0x2c0 [ 7659.537648] kthread+0x1d1/0x200 [ 7659.538966] ? set_kthread_struct+0x70/0x70 [ 7659.540737] ret_from_fork+0x1f/0x30 [ 7659.542293] rcu: Stack dump where RCU GP kthread last ran: [ 7659.544844] Sending NMI from CPU 0 to CPUs 3: [ 7659.546826] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 7659.547789] NMI backtrace for cpu 0 [ 7659.552783] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7659.558077] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7659.562454] Call Trace: [ 7659.563121] [ 7659.563655] ? dump_stack+0xbb/0x10e [ 7659.564804] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7659.566647] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7659.567819] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7659.569052] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7659.574784] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7659.576068] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7659.579174] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7659.581072] ? update_process_times+0x6d/0xc0 [ 7659.582306] ? tick_sched_handle.isra.18+0x27/0x90 [ 7659.583598] ? tick_sched_timer+0x53/0xb0 [ 7659.585643] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7659.588101] ? tick_sched_handle.isra.18+0x90/0x90 [ 7659.589481] ? hrtimer_interrupt+0x128/0x2c0 [ 7659.592262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7659.593866] ? apic_timer_interrupt+0xf/0x20 [ 7659.595867] [ 7659.596630] ? native_safe_halt+0x2e/0x50 [ 7659.598339] ? kvm_wait+0xcd/0xe0 [ 7659.599991] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7659.602027] ? dump_stack+0xf6/0x10e [ 7659.604130] ? do_raw_spin_lock+0x177/0x190 [ 7659.605713] ? _raw_spin_lock+0x21/0x30 [ 7659.606611] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7659.609661] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7659.611239] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7659.613079] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7659.616570] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7659.619701] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7659.623135] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7659.625763] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7659.628555] ? do_wait_intr+0xf0/0xf0 [ 7659.629980] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7659.633544] ? kthread+0x1d1/0x200 [ 7659.634610] ? set_kthread_struct+0x70/0x70 [ 7659.641605] ? ret_from_fork+0x1f/0x30 [ 7660.051770] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7660.059504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7660.102878] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7660.109538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7660.120482] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7660.122861] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7660.126263] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7660.136827] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7660.142111] RAX: 0000000000007d9c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7660.145910] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7660.153160] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7660.156941] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7660.162700] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7660.165928] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7660.170418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7660.173247] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7660.176741] Call Trace: [ 7660.177897] [ 7660.179243] ? show_regs.cold.9+0x22/0x2f [ 7660.184626] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7660.187205] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7660.189416] ? watchdog+0x40/0x40 [ 7660.190963] ? hrtimer_interrupt+0x128/0x2c0 [ 7660.194050] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7660.196403] ? apic_timer_interrupt+0xf/0x20 [ 7660.197443] [ 7660.197927] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7660.199598] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7660.201208] do_raw_spin_lock+0x177/0x190 [ 7660.202733] _raw_spin_lock+0x21/0x30 [ 7660.205242] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7660.207646] process_one_work+0x2c8/0x700 [ 7660.209237] worker_thread+0x296/0x6e0 [ 7660.210928] ? rescuer_thread+0x570/0x570 [ 7660.212891] kthread+0x1d1/0x200 [ 7660.213891] ? set_kthread_struct+0x70/0x70 [ 7660.215802] ret_from_fork+0x1f/0x30 [ 7670.753284] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5645s! [ 7670.756387] Showing busy workqueues and worker pools: [ 7670.760285] workqueue events: flags=0x0 [ 7670.762422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7670.767904] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7670.772399] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7670.775919] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7670.779475] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7670.783414] pending: kfree_rcu_monitor [ 7670.785331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7670.789354] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7670.795105] workqueue events_long: flags=0x0 [ 7670.797373] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7670.800041] pending: lu_objects_flush [obdclass] [ 7670.802305] workqueue events_unbound: flags=0x2 [ 7670.804302] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7670.806057] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7670.809043] workqueue events_power_efficient: flags=0x80 [ 7670.810359] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7670.812154] pending: neigh_periodic_work, check_lifetime [ 7670.814495] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7670.816186] pending: do_cache_clean, neigh_periodic_work [ 7670.817723] workqueue mm_percpu_wq: flags=0x8 [ 7670.819672] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7670.821767] pending: lru_add_drain_per_cpu [ 7670.822891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7670.825356] pending: vmstat_update, lru_add_drain_per_cpu [ 7670.827371] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7670.830502] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7670.833414] workqueue writeback: flags=0x4a [ 7670.835319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7670.838301] pending: wb_workfn, wb_workfn [ 7670.840449] workqueue xprtiod: flags=0xa [ 7670.842332] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7670.845710] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7670.848643] workqueue ipv6_addrconf: flags=0x40008 [ 7670.850395] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7670.854669] pending: addrconf_verify_work [ 7670.858871] workqueue knbd0-recv: flags=0x1a [ 7670.861860] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7670.864342] in-flight: 655:recv_work [ 7670.869828] workqueue ptlrpc_pinger: flags=0x2 [ 7670.871425] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7670.876304] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7670.879601] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5645s workers=4 idle: 36396 38 11188 [ 7670.892670] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7661s workers=2 idle: 658 [ 7680.052480] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7680.056308] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7680.101685] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7680.113698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7680.125082] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7680.129295] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7680.144857] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7680.150760] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7680.153215] RDX: 0000000000003388 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7680.157463] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7680.160957] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7680.165928] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7680.169814] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7680.171947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7680.173337] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7680.180858] Call Trace: [ 7680.181842] [ 7680.182780] ? show_regs.cold.9+0x22/0x2f [ 7680.185627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7680.187899] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7680.189362] ? watchdog+0x40/0x40 [ 7680.190337] ? hrtimer_interrupt+0x128/0x2c0 [ 7680.191852] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7680.193359] ? apic_timer_interrupt+0xf/0x20 [ 7680.194637] [ 7680.195200] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7680.197333] do_raw_spin_lock+0x177/0x190 [ 7680.198902] _raw_spin_lock+0x21/0x30 [ 7680.200916] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7680.204197] ? do_raw_spin_unlock+0x75/0x190 [ 7680.205698] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7680.207670] ? do_wait_intr+0xf0/0xf0 [ 7680.208903] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7680.211412] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7680.213776] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7680.216539] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7680.219417] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7680.223037] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7680.224443] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7680.226223] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7680.227954] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7680.231888] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7680.238826] lmv_intent_open+0x338/0x1150 [lmv] [ 7680.240290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7680.242056] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7680.243794] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7680.245343] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7680.247102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7680.250955] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7680.254880] ? do_raw_spin_unlock+0x75/0x190 [ 7680.258380] ? _raw_spin_unlock+0x12/0x30 [ 7680.262651] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7680.264811] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7680.266706] ll_atomic_open+0x36a/0xd70 [lustre] [ 7680.268754] lookup_open+0x356/0xb40 [ 7680.271716] do_last+0x313/0xfc0 [ 7680.273653] ? nd_jump_root+0xe5/0x160 [ 7680.275619] ? path_init+0x437/0x520 [ 7680.277335] path_openat+0xf7/0x500 [ 7680.278258] ? do_raw_spin_unlock+0x75/0x190 [ 7680.280682] do_filp_open+0x99/0x140 [ 7680.282655] ? getname_flags+0x6e/0x330 [ 7680.284239] ? __check_object_size+0xff/0x256 [ 7680.286837] ? do_raw_spin_unlock+0x75/0x190 [ 7680.289248] ? _raw_spin_unlock+0x12/0x30 [ 7680.291131] do_sys_openat2+0x2b4/0x410 [ 7680.292749] do_sys_open+0x73/0xa0 [ 7680.294133] __x64_sys_openat+0x24/0x30 [ 7680.295869] do_syscall_64+0xc1/0x440 [ 7680.297160] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7680.298587] RIP: 0033:0x7fdc0afbf332 [ 7680.299482] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7680.305062] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7680.307616] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7680.309697] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7680.312285] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7680.313956] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7680.316382] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7684.054289] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 7684.058842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7684.114170] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7684.121119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7684.127592] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7684.130042] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7684.143095] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7684.147321] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7684.150002] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7684.153895] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7684.159117] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7684.166387] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7684.177017] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7684.185471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7684.189341] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7684.195881] Call Trace: [ 7684.197302] [ 7684.198773] ? show_regs.cold.9+0x22/0x2f [ 7684.201299] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7684.207215] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7684.210808] ? watchdog+0x40/0x40 [ 7684.215023] ? hrtimer_interrupt+0x128/0x2c0 [ 7684.218693] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7684.221337] ? apic_timer_interrupt+0xf/0x20 [ 7684.223600] [ 7684.226661] ? native_safe_halt+0x2e/0x50 [ 7684.230064] kvm_wait+0xcd/0xe0 [ 7684.234131] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7684.237754] do_raw_spin_lock+0x177/0x190 [ 7684.240403] _raw_spin_lock+0x21/0x30 [ 7684.243772] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7684.248030] ? do_raw_spin_unlock+0x75/0x190 [ 7684.250290] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7684.253846] ? do_wait_intr+0xf0/0xf0 [ 7684.255406] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7684.258313] mdc_close+0x53b/0xd70 [mdc] [ 7684.260062] lmv_close+0x18d/0x490 [lmv] [ 7684.262190] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7684.265141] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7684.267393] ll_md_close+0x274/0x800 [lustre] [ 7684.269333] ll_file_release+0x128/0xc50 [lustre] [ 7684.271843] __fput+0x10d/0x440 [ 7684.273177] ____fput+0x12/0x20 [ 7684.274617] task_work_run+0xc6/0x110 [ 7684.276410] exit_to_usermode_loop+0x1dd/0x1f0 [ 7684.278416] do_syscall_64+0x3d6/0x440 [ 7684.279880] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7684.282052] RIP: 0033:0x7fdc0afbfcd8 [ 7684.283193] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7684.291226] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7684.295421] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7684.302638] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7684.306123] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7684.309612] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7684.312674] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7688.051103] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7688.055174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7688.080455] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7688.086244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7688.090406] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7688.098456] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7688.100649] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7688.115570] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7688.118212] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7688.123137] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7688.128400] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7688.134020] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7688.138417] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7688.142281] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7688.149650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7688.154163] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7688.159217] Call Trace: [ 7688.160667] [ 7688.161412] ? show_regs.cold.9+0x22/0x2f [ 7688.164592] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7688.166755] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7688.169658] ? watchdog+0x40/0x40 [ 7688.171811] ? hrtimer_interrupt+0x128/0x2c0 [ 7688.176732] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7688.178975] ? apic_timer_interrupt+0xf/0x20 [ 7688.186763] [ 7688.189169] ? native_safe_halt+0x2e/0x50 [ 7688.192831] kvm_wait+0xcd/0xe0 [ 7688.197549] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7688.202247] do_raw_spin_lock+0x177/0x190 [ 7688.208222] _raw_spin_lock+0x21/0x30 [ 7688.214138] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7688.220513] process_one_work+0x2c8/0x700 [ 7688.223387] worker_thread+0x296/0x6e0 [ 7688.225354] ? rescuer_thread+0x570/0x570 [ 7688.227813] kthread+0x1d1/0x200 [ 7688.229089] ? set_kthread_struct+0x70/0x70 [ 7688.230517] ret_from_fork+0x1f/0x30 [ 7696.048022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7696.051111] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7696.081037] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7696.087510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7696.091719] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7696.093289] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7696.100454] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7696.104189] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7696.107568] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7696.111270] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7696.115129] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7696.118958] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7696.122519] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7696.126620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7696.129274] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7696.132442] Call Trace: [ 7696.133533] [ 7696.134481] ? show_regs.cold.9+0x22/0x2f [ 7696.136338] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7696.138388] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7696.140421] ? watchdog+0x40/0x40 [ 7696.141808] ? hrtimer_interrupt+0x128/0x2c0 [ 7696.143968] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7696.147099] ? apic_timer_interrupt+0xf/0x20 [ 7696.149638] [ 7696.150448] ? native_safe_halt+0x2e/0x50 [ 7696.151866] kvm_wait+0xcd/0xe0 [ 7696.155531] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7696.158150] ? dump_stack+0xf6/0x10e [ 7696.160437] do_raw_spin_lock+0x177/0x190 [ 7696.162442] _raw_spin_lock+0x21/0x30 [ 7696.164442] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7696.167327] obd_mod_free+0x46/0x140 [ptlrpc] [ 7696.169583] mdc_commit_open+0x91/0xb0 [mdc] [ 7696.171609] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7696.174263] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7696.177684] after_reply+0xe2c/0x1780 [ptlrpc] [ 7696.180642] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7696.183949] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7696.186701] ? do_wait_intr+0xf0/0xf0 [ 7696.188964] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7696.192292] kthread+0x1d1/0x200 [ 7696.194147] ? set_kthread_struct+0x70/0x70 [ 7696.196355] ret_from_fork+0x1f/0x30 [ 7700.967239] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5676s! [ 7700.972736] Showing busy workqueues and worker pools: [ 7700.976144] workqueue events: flags=0x0 [ 7700.978539] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7700.982622] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7700.986070] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7700.989458] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7700.994057] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7700.997747] pending: kfree_rcu_monitor [ 7700.999967] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7701.002814] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7701.008599] workqueue events_long: flags=0x0 [ 7701.010490] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7701.013439] pending: lu_objects_flush [obdclass] [ 7701.018192] workqueue events_unbound: flags=0x2 [ 7701.021289] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7701.024008] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7701.028598] workqueue events_power_efficient: flags=0x80 [ 7701.030816] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7701.033476] pending: neigh_periodic_work, check_lifetime [ 7701.035700] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7701.038233] pending: do_cache_clean, neigh_periodic_work [ 7701.040320] workqueue mm_percpu_wq: flags=0x8 [ 7701.042118] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7701.044512] pending: lru_add_drain_per_cpu [ 7701.046365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7701.049122] pending: vmstat_update, lru_add_drain_per_cpu [ 7701.051720] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7701.053899] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7701.060493] workqueue writeback: flags=0x4a [ 7701.063386] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7701.066125] pending: wb_workfn, wb_workfn [ 7701.067831] workqueue xprtiod: flags=0xa [ 7701.069179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7701.072011] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7701.075190] workqueue ipv6_addrconf: flags=0x40008 [ 7701.077911] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7701.080882] pending: addrconf_verify_work [ 7701.082868] workqueue knbd0-recv: flags=0x1a [ 7701.084591] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7701.087311] in-flight: 655:recv_work [ 7701.089012] workqueue ptlrpc_pinger: flags=0x2 [ 7701.090986] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7701.093970] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7701.099542] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5676s workers=4 idle: 36396 38 11188 [ 7701.106220] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7691s workers=2 idle: 658 [ 7708.052952] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7708.059599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7708.086196] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7708.093305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7708.101140] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7708.105618] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7708.116866] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7708.122623] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7708.127299] RDX: 0000000000003fa5 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7708.131757] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7708.135290] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7708.138747] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7708.141442] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7708.149602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7708.154057] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7708.156104] Call Trace: [ 7708.157653] [ 7708.158420] ? show_regs.cold.9+0x22/0x2f [ 7708.160665] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7708.165620] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7708.169061] ? watchdog+0x40/0x40 [ 7708.171743] ? hrtimer_interrupt+0x128/0x2c0 [ 7708.173651] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7708.176060] ? apic_timer_interrupt+0xf/0x20 [ 7708.179093] [ 7708.179802] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7708.182103] do_raw_spin_lock+0x177/0x190 [ 7708.187429] _raw_spin_lock+0x21/0x30 [ 7708.188484] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7708.191672] ? do_raw_spin_unlock+0x75/0x190 [ 7708.197404] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7708.200863] ? do_wait_intr+0xf0/0xf0 [ 7708.203227] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7708.207387] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7708.211979] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7708.217344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7708.220038] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7708.222777] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7708.225882] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7708.227492] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7708.236705] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7708.241978] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7708.248243] lmv_intent_open+0x338/0x1150 [lmv] [ 7708.250305] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7708.252012] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7708.254475] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7708.256114] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7708.258298] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7708.261286] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7708.262825] ? do_raw_spin_unlock+0x75/0x190 [ 7708.265996] ? _raw_spin_unlock+0x12/0x30 [ 7708.267413] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7708.269678] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7708.271417] ll_atomic_open+0x36a/0xd70 [lustre] [ 7708.275317] lookup_open+0x356/0xb40 [ 7708.276462] do_last+0x313/0xfc0 [ 7708.279380] ? nd_jump_root+0xe5/0x160 [ 7708.280628] ? path_init+0x437/0x520 [ 7708.281900] path_openat+0xf7/0x500 [ 7708.288341] ? do_raw_spin_unlock+0x75/0x190 [ 7708.291187] do_filp_open+0x99/0x140 [ 7708.293474] ? getname_flags+0x6e/0x330 [ 7708.296474] ? __check_object_size+0xff/0x256 [ 7708.300726] ? do_raw_spin_unlock+0x75/0x190 [ 7708.303437] ? _raw_spin_unlock+0x12/0x30 [ 7708.305267] do_sys_openat2+0x2b4/0x410 [ 7708.306987] do_sys_open+0x73/0xa0 [ 7708.308397] __x64_sys_openat+0x24/0x30 [ 7708.311701] do_syscall_64+0xc1/0x440 [ 7708.314686] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7708.316698] RIP: 0033:0x7fdc0afbf332 [ 7708.319293] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7708.328465] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7708.333737] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7708.337710] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7708.341464] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7708.343917] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7708.346949] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7712.055544] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7712.057328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7712.086276] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7712.101234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7712.109275] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7712.111410] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7712.124474] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7712.127348] RAX: 0000000000006c88 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7712.137708] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7712.149233] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7712.155240] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7712.157482] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7712.160260] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7712.163417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7712.165284] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7712.168225] Call Trace: [ 7712.169315] [ 7712.170052] ? show_regs.cold.9+0x22/0x2f [ 7712.171370] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7712.173208] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7712.175057] ? watchdog+0x40/0x40 [ 7712.179486] ? hrtimer_interrupt+0x128/0x2c0 [ 7712.181182] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7712.183048] ? apic_timer_interrupt+0xf/0x20 [ 7712.184415] [ 7712.185402] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7712.187127] do_raw_spin_lock+0x177/0x190 [ 7712.190359] _raw_spin_lock+0x21/0x30 [ 7712.192880] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7712.195358] ? do_raw_spin_unlock+0x75/0x190 [ 7712.197346] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7712.207089] ? do_wait_intr+0xf0/0xf0 [ 7712.208344] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7712.216844] mdc_close+0x53b/0xd70 [mdc] [ 7712.218823] lmv_close+0x18d/0x490 [lmv] [ 7712.221013] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7712.223494] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7712.225281] ll_md_close+0x274/0x800 [lustre] [ 7712.226704] ll_file_release+0x128/0xc50 [lustre] [ 7712.237663] __fput+0x10d/0x440 [ 7712.238599] ____fput+0x12/0x20 [ 7712.241187] task_work_run+0xc6/0x110 [ 7712.242274] exit_to_usermode_loop+0x1dd/0x1f0 [ 7712.244140] do_syscall_64+0x3d6/0x440 [ 7712.245934] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7712.247969] RIP: 0033:0x7fdc0afbfcd8 [ 7712.249075] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7712.262004] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7712.265092] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7712.267993] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7712.270980] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7712.273806] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7712.276343] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7716.051292] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7716.055133] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7716.119324] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7716.130329] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7716.139193] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7716.142450] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7716.148180] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7716.175376] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7716.183242] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7716.190460] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7716.197599] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7716.208188] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7716.215138] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7716.223192] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7716.227626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7716.238097] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7716.245134] Call Trace: [ 7716.246633] [ 7716.248112] ? show_regs.cold.9+0x22/0x2f [ 7716.253182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7716.254646] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7716.256817] ? watchdog+0x40/0x40 [ 7716.258131] ? hrtimer_interrupt+0x128/0x2c0 [ 7716.260453] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7716.268241] ? apic_timer_interrupt+0xf/0x20 [ 7716.273257] [ 7716.275974] ? native_safe_halt+0x2e/0x50 [ 7716.280085] kvm_wait+0xcd/0xe0 [ 7716.281780] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7716.285578] do_raw_spin_lock+0x177/0x190 [ 7716.287854] _raw_spin_lock+0x21/0x30 [ 7716.292283] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7716.299824] process_one_work+0x2c8/0x700 [ 7716.304121] worker_thread+0x296/0x6e0 [ 7716.311133] ? rescuer_thread+0x570/0x570 [ 7716.312420] kthread+0x1d1/0x200 [ 7716.313352] ? set_kthread_struct+0x70/0x70 [ 7716.318169] ret_from_fork+0x1f/0x30 [ 7728.048804] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 7728.051679] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7728.080791] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7728.086228] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7728.090990] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7728.094262] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7728.102754] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7728.106330] RAX: 0000000000006523 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7728.109716] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7728.112607] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7728.115846] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7728.119484] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7728.123108] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7728.127589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7728.130174] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7728.134053] Call Trace: [ 7728.135309] [ 7728.136207] ? show_regs.cold.9+0x22/0x2f [ 7728.138076] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7728.140438] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7728.142901] ? watchdog+0x40/0x40 [ 7728.144813] ? hrtimer_interrupt+0x128/0x2c0 [ 7728.146861] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7728.150214] ? apic_timer_interrupt+0xf/0x20 [ 7728.152032] [ 7728.153357] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7728.155759] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7728.158573] ? dump_stack+0xf6/0x10e [ 7728.160160] do_raw_spin_lock+0x177/0x190 [ 7728.161988] _raw_spin_lock+0x21/0x30 [ 7728.163361] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7728.165740] obd_mod_free+0x46/0x140 [ptlrpc] [ 7728.168144] mdc_commit_open+0x91/0xb0 [mdc] [ 7728.170028] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7728.173444] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7728.176106] after_reply+0xe2c/0x1780 [ptlrpc] [ 7728.178105] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7728.180493] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7728.182368] ? do_wait_intr+0xf0/0xf0 [ 7728.184120] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7728.186017] kthread+0x1d1/0x200 [ 7728.187284] ? set_kthread_struct+0x70/0x70 [ 7728.188566] ret_from_fork+0x1f/0x30 [ 7731.167936] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5706s! [ 7731.175979] Showing busy workqueues and worker pools: [ 7731.180751] workqueue events: flags=0x0 [ 7731.184071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7731.187873] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7731.193615] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7731.200738] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7731.212436] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7731.216929] pending: kfree_rcu_monitor [ 7731.218969] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7731.227419] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7731.236111] workqueue events_long: flags=0x0 [ 7731.237362] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7731.243949] pending: lu_objects_flush [obdclass] [ 7731.251050] workqueue events_unbound: flags=0x2 [ 7731.257569] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7731.260897] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7731.270720] workqueue events_power_efficient: flags=0x80 [ 7731.277919] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7731.280457] pending: neigh_periodic_work, check_lifetime [ 7731.282471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7731.285400] pending: do_cache_clean, neigh_periodic_work [ 7731.287401] workqueue mm_percpu_wq: flags=0x8 [ 7731.288912] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7731.291259] pending: lru_add_drain_per_cpu [ 7731.292624] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7731.294946] pending: vmstat_update, lru_add_drain_per_cpu [ 7731.297259] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7731.300241] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7731.306438] workqueue writeback: flags=0x4a [ 7731.309024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7731.311182] pending: wb_workfn, wb_workfn [ 7731.315198] workqueue xprtiod: flags=0xa [ 7731.321463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7731.324571] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7731.330510] workqueue ipv6_addrconf: flags=0x40008 [ 7731.331973] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7731.334647] pending: addrconf_verify_work [ 7731.336210] workqueue knbd0-recv: flags=0x1a [ 7731.337950] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7731.342795] in-flight: 655:recv_work [ 7731.344074] workqueue ptlrpc_pinger: flags=0x2 [ 7731.348876] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7731.352218] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7731.355045] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5706s workers=4 idle: 36396 38 11188 [ 7731.357046] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7721s workers=2 idle: 658 [ 7736.052178] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7736.054861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7736.071974] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7736.075856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7736.078323] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7736.080101] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7736.086429] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7736.088414] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7736.090300] RDX: 000000000000274a RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7736.091991] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7736.093871] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7736.095880] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7736.097706] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7736.099889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7736.101521] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7736.103458] Call Trace: [ 7736.104199] [ 7736.104803] ? show_regs.cold.9+0x22/0x2f [ 7736.106211] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7736.107644] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7736.109302] ? watchdog+0x40/0x40 [ 7736.111238] ? hrtimer_interrupt+0x128/0x2c0 [ 7736.112971] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7736.114973] ? apic_timer_interrupt+0xf/0x20 [ 7736.116885] [ 7736.117821] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7736.120551] do_raw_spin_lock+0x177/0x190 [ 7736.122959] _raw_spin_lock+0x21/0x30 [ 7736.124593] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7736.127800] ? do_raw_spin_unlock+0x75/0x190 [ 7736.130633] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7736.133639] ? do_wait_intr+0xf0/0xf0 [ 7736.135355] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7736.138957] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7736.141864] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7736.146876] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7736.149934] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7736.152614] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7736.154919] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7736.157749] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7736.162020] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7736.165399] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7736.168729] lmv_intent_open+0x338/0x1150 [lmv] [ 7736.171918] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7736.174646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7736.177774] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7736.180666] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7736.184672] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7736.188210] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7736.191083] ? do_raw_spin_unlock+0x75/0x190 [ 7736.193177] ? _raw_spin_unlock+0x12/0x30 [ 7736.195580] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7736.198454] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7736.201684] ll_atomic_open+0x36a/0xd70 [lustre] [ 7736.203426] lookup_open+0x356/0xb40 [ 7736.204673] do_last+0x313/0xfc0 [ 7736.207667] ? nd_jump_root+0xe5/0x160 [ 7736.209039] ? path_init+0x437/0x520 [ 7736.211564] path_openat+0xf7/0x500 [ 7736.212784] ? do_raw_spin_unlock+0x75/0x190 [ 7736.215678] do_filp_open+0x99/0x140 [ 7736.217475] ? getname_flags+0x6e/0x330 [ 7736.219617] ? __check_object_size+0xff/0x256 [ 7736.221521] ? do_raw_spin_unlock+0x75/0x190 [ 7736.225365] ? _raw_spin_unlock+0x12/0x30 [ 7736.226866] do_sys_openat2+0x2b4/0x410 [ 7736.229480] do_sys_open+0x73/0xa0 [ 7736.231530] __x64_sys_openat+0x24/0x30 [ 7736.233499] do_syscall_64+0xc1/0x440 [ 7736.235825] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7736.240577] RIP: 0033:0x7fdc0afbf332 [ 7736.243997] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7736.254702] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7736.258543] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7736.262046] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7736.266795] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7736.274667] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7736.276356] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7740.054314] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7740.058210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7740.079620] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7740.087784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7740.091825] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7740.094253] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7740.102448] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7740.106227] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7740.108493] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7740.111134] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7740.113626] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7740.116648] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7740.119439] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7740.122546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7740.124874] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7740.128181] Call Trace: [ 7740.129275] [ 7740.130434] ? show_regs.cold.9+0x22/0x2f [ 7740.132643] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7740.134684] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7740.136760] ? watchdog+0x40/0x40 [ 7740.138191] ? hrtimer_interrupt+0x128/0x2c0 [ 7740.139577] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7740.141584] ? apic_timer_interrupt+0xf/0x20 [ 7740.143303] [ 7740.144216] ? native_safe_halt+0x2e/0x50 [ 7740.145860] kvm_wait+0xcd/0xe0 [ 7740.146890] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7740.148531] do_raw_spin_lock+0x177/0x190 [ 7740.150339] _raw_spin_lock+0x21/0x30 [ 7740.151913] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7740.154483] ? do_raw_spin_unlock+0x75/0x190 [ 7740.156141] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7740.158412] ? do_wait_intr+0xf0/0xf0 [ 7740.160185] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7740.162480] mdc_close+0x53b/0xd70 [mdc] [ 7740.164273] lmv_close+0x18d/0x490 [lmv] [ 7740.166044] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7740.169315] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7740.171491] ll_md_close+0x274/0x800 [lustre] [ 7740.174192] ll_file_release+0x128/0xc50 [lustre] [ 7740.178197] __fput+0x10d/0x440 [ 7740.183207] ____fput+0x12/0x20 [ 7740.184263] task_work_run+0xc6/0x110 [ 7740.185774] exit_to_usermode_loop+0x1dd/0x1f0 [ 7740.187225] do_syscall_64+0x3d6/0x440 [ 7740.189160] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7740.191582] RIP: 0033:0x7fdc0afbfcd8 [ 7740.194554] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7740.201402] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7740.204854] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7740.207850] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7740.210560] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7740.215435] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7740.218999] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7744.051447] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7744.055646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7744.079460] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7744.086266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7744.091170] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7744.093584] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7744.095383] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7744.102000] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7744.104379] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7744.110077] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7744.112585] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7744.115490] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7744.118086] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7744.123460] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7744.126785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7744.132225] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7744.135096] Call Trace: [ 7744.138510] [ 7744.139152] ? show_regs.cold.9+0x22/0x2f [ 7744.140229] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7744.142752] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7744.144566] ? watchdog+0x40/0x40 [ 7744.146086] ? hrtimer_interrupt+0x128/0x2c0 [ 7744.147475] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7744.149436] ? apic_timer_interrupt+0xf/0x20 [ 7744.151146] [ 7744.151772] ? native_safe_halt+0x2e/0x50 [ 7744.156549] kvm_wait+0xcd/0xe0 [ 7744.158088] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7744.160062] do_raw_spin_lock+0x177/0x190 [ 7744.161207] _raw_spin_lock+0x21/0x30 [ 7744.165076] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7744.168522] process_one_work+0x2c8/0x700 [ 7744.170214] worker_thread+0x296/0x6e0 [ 7744.171526] ? rescuer_thread+0x570/0x570 [ 7744.172986] kthread+0x1d1/0x200 [ 7744.174034] ? set_kthread_struct+0x70/0x70 [ 7744.175644] ret_from_fork+0x1f/0x30 [ 7756.049124] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7756.052309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7756.075122] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7756.080244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7756.084449] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7756.087339] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7756.095427] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7756.099442] RAX: 0000000000005e4f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7756.103302] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7756.106961] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7756.110517] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7756.114565] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7756.118503] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7756.122884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7756.125437] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7756.129204] Call Trace: [ 7756.130249] [ 7756.131256] ? show_regs.cold.9+0x22/0x2f [ 7756.133237] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7756.135344] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7756.137529] ? watchdog+0x40/0x40 [ 7756.139265] ? hrtimer_interrupt+0x128/0x2c0 [ 7756.141225] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7756.143646] ? apic_timer_interrupt+0xf/0x20 [ 7756.145333] [ 7756.146331] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7756.148635] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7756.150609] ? dump_stack+0xf6/0x10e [ 7756.152187] do_raw_spin_lock+0x177/0x190 [ 7756.153631] _raw_spin_lock+0x21/0x30 [ 7756.155094] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7756.157051] obd_mod_free+0x46/0x140 [ptlrpc] [ 7756.158983] mdc_commit_open+0x91/0xb0 [mdc] [ 7756.160500] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7756.162742] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7756.165027] after_reply+0xe2c/0x1780 [ptlrpc] [ 7756.166906] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7756.168974] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7756.171003] ? do_wait_intr+0xf0/0xf0 [ 7756.172554] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7756.174653] kthread+0x1d1/0x200 [ 7756.176635] ? set_kthread_struct+0x70/0x70 [ 7756.178270] ret_from_fork+0x1f/0x30 [ 7761.378277] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5736s! [ 7761.383949] Showing busy workqueues and worker pools: [ 7761.387541] workqueue events: flags=0x0 [ 7761.389892] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7761.393665] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7761.397669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7761.407033] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7761.415790] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7761.424488] pending: kfree_rcu_monitor [ 7761.430509] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7761.437045] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7761.444851] workqueue events_long: flags=0x0 [ 7761.447748] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7761.451629] pending: lu_objects_flush [obdclass] [ 7761.454273] workqueue events_unbound: flags=0x2 [ 7761.456878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7761.461014] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7761.470075] workqueue events_power_efficient: flags=0x80 [ 7761.475376] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7761.478180] pending: neigh_periodic_work, check_lifetime [ 7761.485600] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7761.488940] pending: do_cache_clean, neigh_periodic_work [ 7761.500586] workqueue mm_percpu_wq: flags=0x8 [ 7761.504542] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7761.509892] pending: lru_add_drain_per_cpu [ 7761.512152] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7761.516316] pending: vmstat_update, lru_add_drain_per_cpu [ 7761.518908] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7761.521609] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7761.525735] workqueue writeback: flags=0x4a [ 7761.527693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7761.530868] pending: wb_workfn, wb_workfn [ 7761.533093] workqueue xprtiod: flags=0xa [ 7761.534568] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7761.537633] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7761.543620] workqueue ipv6_addrconf: flags=0x40008 [ 7761.544834] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7761.547373] pending: addrconf_verify_work [ 7761.548449] workqueue knbd0-recv: flags=0x1a [ 7761.549685] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7761.551225] in-flight: 655:recv_work [ 7761.552588] workqueue ptlrpc_pinger: flags=0x2 [ 7761.554678] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7761.557032] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7761.559855] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5736s workers=4 idle: 36396 38 11188 [ 7761.563468] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7752s workers=2 idle: 658 [ 7764.052023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7764.059740] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7764.086809] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7764.095813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7764.102337] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7764.106351] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7764.116346] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7764.123204] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7764.125042] RDX: 0000000000006002 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7764.127714] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7764.130423] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7764.133373] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7764.136090] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7764.139258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7764.142357] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7764.144635] Call Trace: [ 7764.145511] [ 7764.146287] ? show_regs.cold.9+0x22/0x2f [ 7764.147830] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7764.149622] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7764.151215] ? watchdog+0x40/0x40 [ 7764.152983] ? hrtimer_interrupt+0x128/0x2c0 [ 7764.154549] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7764.156643] ? apic_timer_interrupt+0xf/0x20 [ 7764.158247] [ 7764.159359] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7764.161481] do_raw_spin_lock+0x177/0x190 [ 7764.163291] _raw_spin_lock+0x21/0x30 [ 7764.164859] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7764.167373] ? do_raw_spin_unlock+0x75/0x190 [ 7764.168567] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7764.170761] ? do_wait_intr+0xf0/0xf0 [ 7764.172316] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7764.174480] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7764.176571] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7764.179128] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7764.181830] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7764.184388] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7764.186356] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7764.188340] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7764.190803] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7764.193835] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7764.196249] lmv_intent_open+0x338/0x1150 [lmv] [ 7764.198262] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7764.200960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7764.203581] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7764.205600] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7764.207781] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7764.210189] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7764.212410] ? do_raw_spin_unlock+0x75/0x190 [ 7764.214319] ? _raw_spin_unlock+0x12/0x30 [ 7764.216392] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7764.218817] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7764.221316] ll_atomic_open+0x36a/0xd70 [lustre] [ 7764.223332] lookup_open+0x356/0xb40 [ 7764.225109] do_last+0x313/0xfc0 [ 7764.226903] ? nd_jump_root+0xe5/0x160 [ 7764.228466] ? path_init+0x437/0x520 [ 7764.230144] path_openat+0xf7/0x500 [ 7764.231837] ? do_raw_spin_unlock+0x75/0x190 [ 7764.233589] do_filp_open+0x99/0x140 [ 7764.235357] ? getname_flags+0x6e/0x330 [ 7764.237040] ? __check_object_size+0xff/0x256 [ 7764.238855] ? do_raw_spin_unlock+0x75/0x190 [ 7764.240410] ? _raw_spin_unlock+0x12/0x30 [ 7764.241391] do_sys_openat2+0x2b4/0x410 [ 7764.242763] do_sys_open+0x73/0xa0 [ 7764.244232] __x64_sys_openat+0x24/0x30 [ 7764.246074] do_syscall_64+0xc1/0x440 [ 7764.247854] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7764.250136] RIP: 0033:0x7fdc0afbf332 [ 7764.251549] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7764.258728] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7764.262397] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7764.265946] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7764.269488] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7764.272464] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7764.275644] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7768.056732] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7768.064348] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7768.098134] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7768.103066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7768.106135] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7768.108802] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7768.118004] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7768.120300] RAX: 0000000000007f24 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7768.123163] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7768.125780] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7768.129064] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7768.132421] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7768.135957] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7768.139794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7768.144885] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7768.147446] Call Trace: [ 7768.149877] [ 7768.150410] ? show_regs.cold.9+0x22/0x2f [ 7768.151474] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7768.152582] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7768.155068] ? watchdog+0x40/0x40 [ 7768.156426] ? hrtimer_interrupt+0x128/0x2c0 [ 7768.158409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7768.159995] ? apic_timer_interrupt+0xf/0x20 [ 7768.161192] [ 7768.161926] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7768.164170] do_raw_spin_lock+0x177/0x190 [ 7768.166012] _raw_spin_lock+0x21/0x30 [ 7768.167525] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7768.171417] ? do_raw_spin_unlock+0x75/0x190 [ 7768.172761] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7768.175446] ? do_wait_intr+0xf0/0xf0 [ 7768.177008] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7768.179166] mdc_close+0x53b/0xd70 [mdc] [ 7768.180769] lmv_close+0x18d/0x490 [lmv] [ 7768.182564] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7768.185044] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7768.186793] ll_md_close+0x274/0x800 [lustre] [ 7768.188343] ll_file_release+0x128/0xc50 [lustre] [ 7768.189807] __fput+0x10d/0x440 [ 7768.194537] ____fput+0x12/0x20 [ 7768.195750] task_work_run+0xc6/0x110 [ 7768.197136] exit_to_usermode_loop+0x1dd/0x1f0 [ 7768.202884] do_syscall_64+0x3d6/0x440 [ 7768.204928] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7768.207021] RIP: 0033:0x7fdc0afbfcd8 [ 7768.212444] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7768.226780] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7768.230399] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7768.233579] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7768.236378] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7768.241385] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7768.245227] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7772.054018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7772.063127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7772.116196] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7772.121565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7772.126296] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7772.131557] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7772.136269] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7772.155545] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7772.158574] RAX: 0000000000001d4b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7772.165149] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7772.173200] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7772.178621] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7772.186671] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7772.189078] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7772.196266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7772.198713] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7772.201775] Call Trace: [ 7772.205803] [ 7772.206771] ? show_regs.cold.9+0x22/0x2f [ 7772.209702] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7772.211922] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7772.213871] ? watchdog+0x40/0x40 [ 7772.215302] ? hrtimer_interrupt+0x128/0x2c0 [ 7772.217082] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7772.219299] ? apic_timer_interrupt+0xf/0x20 [ 7772.221266] [ 7772.222245] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7772.224218] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7772.228384] do_raw_spin_lock+0x177/0x190 [ 7772.229677] _raw_spin_lock+0x21/0x30 [ 7772.234203] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7772.237733] process_one_work+0x2c8/0x700 [ 7772.240996] worker_thread+0x296/0x6e0 [ 7772.242161] ? rescuer_thread+0x570/0x570 [ 7772.243473] kthread+0x1d1/0x200 [ 7772.247689] ? set_kthread_struct+0x70/0x70 [ 7772.251880] ret_from_fork+0x1f/0x30 [ 7788.048332] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 7788.055327] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7788.072081] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7788.080940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7788.089848] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7788.091900] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7788.098676] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7788.101534] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7788.105518] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7788.110733] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7788.113357] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7788.115446] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7788.117707] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7788.121352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7788.123527] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7788.125821] Call Trace: [ 7788.126770] [ 7788.127385] ? show_regs.cold.9+0x22/0x2f [ 7788.128657] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7788.130163] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7788.131408] ? watchdog+0x40/0x40 [ 7788.132209] ? hrtimer_interrupt+0x128/0x2c0 [ 7788.133269] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7788.134870] ? apic_timer_interrupt+0xf/0x20 [ 7788.136585] [ 7788.137398] ? native_safe_halt+0x2e/0x50 [ 7788.138757] kvm_wait+0xcd/0xe0 [ 7788.139991] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7788.142867] ? dump_stack+0xf6/0x10e [ 7788.143948] do_raw_spin_lock+0x177/0x190 [ 7788.146323] _raw_spin_lock+0x21/0x30 [ 7788.147999] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7788.149759] obd_mod_free+0x46/0x140 [ptlrpc] [ 7788.151606] mdc_commit_open+0x91/0xb0 [mdc] [ 7788.152926] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7788.154646] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7788.157301] after_reply+0xe2c/0x1780 [ptlrpc] [ 7788.160075] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7788.164053] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7788.165253] ? do_wait_intr+0xf0/0xf0 [ 7788.167585] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7788.170570] kthread+0x1d1/0x200 [ 7788.172016] ? set_kthread_struct+0x70/0x70 [ 7788.173737] ret_from_fork+0x1f/0x30 [ 7791.583030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5766s! [ 7791.586229] Showing busy workqueues and worker pools: [ 7791.588347] workqueue events: flags=0x0 [ 7791.589664] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7791.598951] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7791.602537] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7791.607134] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7791.616310] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7791.621720] pending: kfree_rcu_monitor [ 7791.623737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7791.627176] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7791.643155] workqueue events_long: flags=0x0 [ 7791.644462] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7791.648390] pending: lu_objects_flush [obdclass] [ 7791.651648] workqueue events_unbound: flags=0x2 [ 7791.653398] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7791.659262] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7791.665890] workqueue events_power_efficient: flags=0x80 [ 7791.667301] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7791.670302] pending: neigh_periodic_work, check_lifetime [ 7791.672337] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7791.674719] pending: do_cache_clean, neigh_periodic_work [ 7791.677919] workqueue mm_percpu_wq: flags=0x8 [ 7791.680155] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7791.686014] pending: lru_add_drain_per_cpu [ 7791.687830] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7791.690508] pending: vmstat_update, lru_add_drain_per_cpu [ 7791.693949] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7791.697316] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7791.699640] workqueue writeback: flags=0x4a [ 7791.701327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7791.705206] pending: wb_workfn, wb_workfn [ 7791.707656] workqueue xprtiod: flags=0xa [ 7791.709973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7791.716661] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7791.722993] workqueue ipv6_addrconf: flags=0x40008 [ 7791.726635] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7791.730535] pending: addrconf_verify_work [ 7791.734820] workqueue knbd0-recv: flags=0x1a [ 7791.736585] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7791.739852] in-flight: 655:recv_work [ 7791.741856] workqueue ptlrpc_pinger: flags=0x2 [ 7791.743867] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7791.746373] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7791.749209] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5766s workers=4 idle: 36396 38 11188 [ 7791.757884] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7782s workers=2 idle: 658 [ 7792.053515] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7792.060006] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7792.094224] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7792.099325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7792.103821] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7792.105946] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7792.113200] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7792.116149] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7792.118944] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7792.122021] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7792.125108] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7792.128109] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7792.131017] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7792.134135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7792.136426] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7792.139457] Call Trace: [ 7792.140535] [ 7792.141390] ? show_regs.cold.9+0x22/0x2f [ 7792.142886] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7792.144271] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7792.145698] ? watchdog+0x40/0x40 [ 7792.146970] ? hrtimer_interrupt+0x128/0x2c0 [ 7792.148704] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7792.150677] ? apic_timer_interrupt+0xf/0x20 [ 7792.152533] [ 7792.153144] ? native_safe_halt+0x2e/0x50 [ 7792.154700] kvm_wait+0xcd/0xe0 [ 7792.155972] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7792.158219] do_raw_spin_lock+0x177/0x190 [ 7792.159739] _raw_spin_lock+0x21/0x30 [ 7792.161143] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7792.163646] ? do_raw_spin_unlock+0x75/0x190 [ 7792.165549] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7792.167526] ? do_wait_intr+0xf0/0xf0 [ 7792.170096] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7792.172218] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7792.174632] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7792.177302] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7792.179530] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7792.181799] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7792.183859] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7792.185881] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7792.189284] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7792.192662] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7792.195455] lmv_intent_open+0x338/0x1150 [lmv] [ 7792.196980] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7792.198819] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7792.200928] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7792.203155] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7792.205524] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7792.207822] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7792.214649] ? do_raw_spin_unlock+0x75/0x190 [ 7792.219080] ? _raw_spin_unlock+0x12/0x30 [ 7792.222979] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7792.227288] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7792.229486] ll_atomic_open+0x36a/0xd70 [lustre] [ 7792.231566] lookup_open+0x356/0xb40 [ 7792.233419] do_last+0x313/0xfc0 [ 7792.234730] ? nd_jump_root+0xe5/0x160 [ 7792.236739] ? path_init+0x437/0x520 [ 7792.238706] path_openat+0xf7/0x500 [ 7792.240316] ? do_raw_spin_unlock+0x75/0x190 [ 7792.241890] do_filp_open+0x99/0x140 [ 7792.244050] ? getname_flags+0x6e/0x330 [ 7792.246363] ? __check_object_size+0xff/0x256 [ 7792.248446] ? do_raw_spin_unlock+0x75/0x190 [ 7792.250848] ? _raw_spin_unlock+0x12/0x30 [ 7792.252364] do_sys_openat2+0x2b4/0x410 [ 7792.254261] do_sys_open+0x73/0xa0 [ 7792.255386] __x64_sys_openat+0x24/0x30 [ 7792.256963] do_syscall_64+0xc1/0x440 [ 7792.258667] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7792.260911] RIP: 0033:0x7fdc0afbf332 [ 7792.262214] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7792.269917] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7792.273188] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7792.276588] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7792.279292] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7792.282408] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7792.285075] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7796.054110] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7796.057732] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7796.078059] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7796.082852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7796.086735] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7796.089765] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7796.099121] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7796.102919] RAX: 0000000000001b36 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7796.105939] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7796.109764] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7796.113873] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7796.117228] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7796.120499] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7796.124000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7796.126697] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7796.130972] Call Trace: [ 7796.131990] [ 7796.132890] ? show_regs.cold.9+0x22/0x2f [ 7796.134819] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7796.137563] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7796.139552] ? watchdog+0x40/0x40 [ 7796.140860] ? hrtimer_interrupt+0x128/0x2c0 [ 7796.142922] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7796.145457] ? apic_timer_interrupt+0xf/0x20 [ 7796.147325] [ 7796.148267] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7796.150490] do_raw_spin_lock+0x177/0x190 [ 7796.152446] _raw_spin_lock+0x21/0x30 [ 7796.154326] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7796.156692] ? do_raw_spin_unlock+0x75/0x190 [ 7796.158251] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7796.160399] ? do_wait_intr+0xf0/0xf0 [ 7796.162337] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7796.164908] mdc_close+0x53b/0xd70 [mdc] [ 7796.167811] lmv_close+0x18d/0x490 [lmv] [ 7796.170469] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7796.173385] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7796.175678] ll_md_close+0x274/0x800 [lustre] [ 7796.177700] ll_file_release+0x128/0xc50 [lustre] [ 7796.180139] __fput+0x10d/0x440 [ 7796.181706] ____fput+0x12/0x20 [ 7796.183154] task_work_run+0xc6/0x110 [ 7796.184813] exit_to_usermode_loop+0x1dd/0x1f0 [ 7796.186726] do_syscall_64+0x3d6/0x440 [ 7796.188753] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7796.191135] RIP: 0033:0x7fdc0afbfcd8 [ 7796.192668] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7796.201559] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7796.204230] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7796.206943] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7796.213065] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7796.216533] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7796.222474] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7800.051009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7800.053699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7800.076353] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7800.081204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7800.084354] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7800.087071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7800.089329] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7800.096025] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7800.098799] RAX: 0000000000004d3c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7800.101745] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7800.104577] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7800.107068] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7800.109365] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7800.111395] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7800.113358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7800.114942] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7800.117544] Call Trace: [ 7800.118481] [ 7800.119087] ? show_regs.cold.9+0x22/0x2f [ 7800.120408] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7800.121536] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7800.122782] ? watchdog+0x40/0x40 [ 7800.123799] ? hrtimer_interrupt+0x128/0x2c0 [ 7800.125184] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7800.126820] ? apic_timer_interrupt+0xf/0x20 [ 7800.128431] [ 7800.129214] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7800.132135] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7800.134018] do_raw_spin_lock+0x177/0x190 [ 7800.135238] _raw_spin_lock+0x21/0x30 [ 7800.136396] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7800.138184] process_one_work+0x2c8/0x700 [ 7800.139530] worker_thread+0x296/0x6e0 [ 7800.140641] ? rescuer_thread+0x570/0x570 [ 7800.141738] kthread+0x1d1/0x200 [ 7800.142621] ? set_kthread_struct+0x70/0x70 [ 7800.145442] ret_from_fork+0x1f/0x30 [ 7816.048039] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 7816.051220] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7816.073235] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7816.079944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7816.083551] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7816.086008] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7816.095592] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7816.102686] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7816.112244] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7816.121188] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7816.125521] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7816.135794] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7816.145111] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7816.150091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7816.154163] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7816.158241] Call Trace: [ 7816.160826] [ 7816.162896] ? show_regs.cold.9+0x22/0x2f [ 7816.165329] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7816.168196] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7816.170202] ? watchdog+0x40/0x40 [ 7816.172268] ? hrtimer_interrupt+0x128/0x2c0 [ 7816.173363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7816.174844] ? apic_timer_interrupt+0xf/0x20 [ 7816.176069] [ 7816.177807] ? native_safe_halt+0x2e/0x50 [ 7816.179786] kvm_wait+0xcd/0xe0 [ 7816.180616] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7816.182047] ? dump_stack+0xf6/0x10e [ 7816.183721] do_raw_spin_lock+0x177/0x190 [ 7816.186008] _raw_spin_lock+0x21/0x30 [ 7816.187989] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7816.190954] obd_mod_free+0x46/0x140 [ptlrpc] [ 7816.193998] mdc_commit_open+0x91/0xb0 [mdc] [ 7816.196353] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7816.200250] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7816.204272] after_reply+0xe2c/0x1780 [ptlrpc] [ 7816.207409] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7816.210699] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7816.219204] ? do_wait_intr+0xf0/0xf0 [ 7816.227245] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7816.233562] kthread+0x1d1/0x200 [ 7816.236449] ? set_kthread_struct+0x70/0x70 [ 7816.238511] ret_from_fork+0x1f/0x30 [ 7820.052008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7820.056572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7820.087471] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7820.095910] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7820.101741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7820.107390] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7820.119518] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7820.122785] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7820.125921] RDX: 000000000000038c RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7820.130361] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7820.135058] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7820.139182] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7820.142361] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7820.147724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7820.152882] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7820.156905] Call Trace: [ 7820.157974] [ 7820.159789] ? show_regs.cold.9+0x22/0x2f [ 7820.161184] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7820.166427] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7820.169600] ? watchdog+0x40/0x40 [ 7820.173084] ? hrtimer_interrupt+0x128/0x2c0 [ 7820.178502] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7820.181881] ? apic_timer_interrupt+0xf/0x20 [ 7820.191010] [ 7820.191813] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7820.197480] do_raw_spin_lock+0x177/0x190 [ 7820.199791] _raw_spin_lock+0x21/0x30 [ 7820.204966] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7820.208430] ? do_raw_spin_unlock+0x75/0x190 [ 7820.211707] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7820.221716] ? do_wait_intr+0xf0/0xf0 [ 7820.222889] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7820.230777] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7820.234754] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7820.238390] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7820.245409] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7820.248743] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7820.251018] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7820.253095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7820.258182] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7820.260056] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7820.264333] lmv_intent_open+0x338/0x1150 [lmv] [ 7820.266444] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7820.270486] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7820.277472] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7820.280048] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7820.282069] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7820.283913] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7820.285082] ? do_raw_spin_unlock+0x75/0x190 [ 7820.286212] ? _raw_spin_unlock+0x12/0x30 [ 7820.289066] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7820.291786] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7820.295392] ll_atomic_open+0x36a/0xd70 [lustre] [ 7820.297452] lookup_open+0x356/0xb40 [ 7820.299268] do_last+0x313/0xfc0 [ 7820.301679] ? nd_jump_root+0xe5/0x160 [ 7820.304906] ? path_init+0x437/0x520 [ 7820.307110] path_openat+0xf7/0x500 [ 7820.307954] ? do_raw_spin_unlock+0x75/0x190 [ 7820.309769] do_filp_open+0x99/0x140 [ 7820.311285] ? getname_flags+0x6e/0x330 [ 7820.312740] ? __check_object_size+0xff/0x256 [ 7820.314298] ? do_raw_spin_unlock+0x75/0x190 [ 7820.315845] ? _raw_spin_unlock+0x12/0x30 [ 7820.318747] do_sys_openat2+0x2b4/0x410 [ 7820.319828] do_sys_open+0x73/0xa0 [ 7820.320774] __x64_sys_openat+0x24/0x30 [ 7820.324317] do_syscall_64+0xc1/0x440 [ 7820.326444] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7820.333247] RIP: 0033:0x7fdc0afbf332 [ 7820.335872] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7820.349808] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7820.357521] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7820.361476] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7820.368928] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7820.376312] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7820.379794] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7821.791102] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5796s! [ 7821.794188] Showing busy workqueues and worker pools: [ 7821.796310] workqueue events: flags=0x0 [ 7821.798053] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7821.801094] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7821.803906] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7821.806818] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7821.811095] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7821.814021] pending: kfree_rcu_monitor [ 7821.815941] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7821.818922] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7821.824185] workqueue events_long: flags=0x0 [ 7821.825957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7821.828765] pending: lu_objects_flush [obdclass] [ 7821.831258] workqueue events_unbound: flags=0x2 [ 7821.833430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7821.836031] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7821.840270] workqueue events_power_efficient: flags=0x80 [ 7821.842741] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7821.845648] pending: neigh_periodic_work, check_lifetime [ 7821.848189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7821.851248] pending: do_cache_clean, neigh_periodic_work [ 7821.853592] workqueue mm_percpu_wq: flags=0x8 [ 7821.855109] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7821.858102] pending: lru_add_drain_per_cpu [ 7821.860116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7821.863198] pending: vmstat_update, lru_add_drain_per_cpu [ 7821.865808] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7821.879181] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7821.882429] workqueue writeback: flags=0x4a [ 7821.884315] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7821.886363] pending: wb_workfn, wb_workfn [ 7821.887598] workqueue xprtiod: flags=0xa [ 7821.889670] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7821.892726] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7821.896484] workqueue ipv6_addrconf: flags=0x40008 [ 7821.898795] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7821.902176] pending: addrconf_verify_work [ 7821.903884] workqueue knbd0-recv: flags=0x1a [ 7821.906104] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7821.912196] in-flight: 655:recv_work [ 7821.914465] workqueue ptlrpc_pinger: flags=0x2 [ 7821.917145] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7821.920712] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7821.923632] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5796s workers=4 idle: 36396 38 11188 [ 7821.927469] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7812s workers=2 idle: 658 [ 7824.058451] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7824.067500] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7824.113884] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7824.120626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7824.126028] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7824.129352] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7824.146826] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7824.151667] RAX: 0000000000005f88 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7824.160367] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7824.164659] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7824.172866] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7824.176215] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7824.180569] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7824.184824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7824.187406] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7824.190000] Call Trace: [ 7824.190657] [ 7824.191578] ? show_regs.cold.9+0x22/0x2f [ 7824.195736] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7824.198785] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7824.205257] ? watchdog+0x40/0x40 [ 7824.206507] ? hrtimer_interrupt+0x128/0x2c0 [ 7824.208658] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7824.210240] ? apic_timer_interrupt+0xf/0x20 [ 7824.216440] [ 7824.217403] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7824.223132] do_raw_spin_lock+0x177/0x190 [ 7824.224673] _raw_spin_lock+0x21/0x30 [ 7824.226692] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7824.232018] ? do_raw_spin_unlock+0x75/0x190 [ 7824.235799] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7824.242055] ? do_wait_intr+0xf0/0xf0 [ 7824.246114] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7824.248891] mdc_close+0x53b/0xd70 [mdc] [ 7824.250788] lmv_close+0x18d/0x490 [lmv] [ 7824.255908] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7824.263213] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7824.266548] ll_md_close+0x274/0x800 [lustre] [ 7824.270961] ll_file_release+0x128/0xc50 [lustre] [ 7824.275069] __fput+0x10d/0x440 [ 7824.276833] ____fput+0x12/0x20 [ 7824.278517] task_work_run+0xc6/0x110 [ 7824.280713] exit_to_usermode_loop+0x1dd/0x1f0 [ 7824.287096] do_syscall_64+0x3d6/0x440 [ 7824.289199] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7824.291784] RIP: 0033:0x7fdc0afbfcd8 [ 7824.293663] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7824.302180] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7824.305198] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7824.308141] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7824.311015] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7824.313752] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7824.316732] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7828.051028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7828.055031] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7828.089458] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7828.095249] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7828.099948] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7828.102994] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7828.105426] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7828.114167] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7828.117866] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7828.121218] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7828.124361] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7828.128156] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7828.131491] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7828.140127] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7828.148079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7828.155341] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7828.159591] Call Trace: [ 7828.160886] [ 7828.161935] ? show_regs.cold.9+0x22/0x2f [ 7828.164069] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7828.166274] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7828.168383] ? watchdog+0x40/0x40 [ 7828.170028] ? hrtimer_interrupt+0x128/0x2c0 [ 7828.173821] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7828.176128] ? apic_timer_interrupt+0xf/0x20 [ 7828.177512] [ 7828.178388] ? native_safe_halt+0x2e/0x50 [ 7828.181975] kvm_wait+0xcd/0xe0 [ 7828.183456] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7828.185637] do_raw_spin_lock+0x177/0x190 [ 7828.187348] _raw_spin_lock+0x21/0x30 [ 7828.188741] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7828.191186] process_one_work+0x2c8/0x700 [ 7828.192874] worker_thread+0x296/0x6e0 [ 7828.194443] ? rescuer_thread+0x570/0x570 [ 7828.196186] kthread+0x1d1/0x200 [ 7828.197433] ? set_kthread_struct+0x70/0x70 [ 7828.199114] ret_from_fork+0x1f/0x30 [ 7839.487028] rcu: INFO: rcu_sched self-detected stall on CPU [ 7839.489289] rcu: 0-...!: (3627487 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 7839.494043] (t=5820144 jiffies g=270761 q=4039) [ 7839.495431] rcu: rcu_sched kthread starved for 5815076 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 7839.498624] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7839.502179] rcu: RCU grace-period kthread stack dump: [ 7839.504106] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7839.507122] Call Trace: [ 7839.507954] __schedule+0x351/0xcb0 [ 7839.508948] schedule+0xc0/0x180 [ 7839.509700] schedule_timeout+0xb4/0x190 [ 7839.510951] ? do_raw_spin_unlock+0x75/0x190 [ 7839.512101] ? __next_timer_interrupt+0x160/0x160 [ 7839.513498] ? prepare_to_swait_event+0xf3/0x140 [ 7839.514687] rcu_gp_kthread+0x8dd/0x15b0 [ 7839.516210] ? force_qs_rnp+0x2c0/0x2c0 [ 7839.517131] kthread+0x1d1/0x200 [ 7839.518071] ? set_kthread_struct+0x70/0x70 [ 7839.519479] ret_from_fork+0x1f/0x30 [ 7839.520689] rcu: Stack dump where RCU GP kthread last ran: [ 7839.522268] Sending NMI from CPU 0 to CPUs 1: [ 7839.524270] NMI backtrace for cpu 1 [ 7839.524275] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7839.524277] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7839.524279] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7839.524282] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7839.524283] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7839.524286] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 [ 7839.524287] RAX: 00000000000005c5 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7839.524289] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7839.524290] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7839.524291] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7839.524293] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7839.524294] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7839.524295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7839.524296] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7839.524297] Call Trace: [ 7839.524298] [ 7839.524299] ? show_regs.cold.9+0x22/0x2f [ 7839.524300] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7839.524301] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7839.524302] ? nmi_handle+0xbd/0x210 [ 7839.524303] ? default_do_nmi+0x4b/0x1b0 [ 7839.524304] ? do_nmi+0x280/0x380 [ 7839.524305] ? end_repeat_nmi+0x16/0x1f [ 7839.524306] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7839.524307] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7839.524308] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7839.524309] [ 7839.524310] do_raw_spin_lock+0x177/0x190 [ 7839.524311] _raw_spin_lock+0x21/0x30 [ 7839.524312] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7839.524313] process_one_work+0x2c8/0x700 [ 7839.524314] worker_thread+0x296/0x6e0 [ 7839.524315] ? rescuer_thread+0x570/0x570 [ 7839.524316] kthread+0x1d1/0x200 [ 7839.524317] ? set_kthread_struct+0x70/0x70 [ 7839.524318] ret_from_fork+0x1f/0x30 [ 7839.524684] NMI backtrace for cpu 0 [ 7839.590497] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7839.593976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7839.596602] Call Trace: [ 7839.597274] [ 7839.598152] ? dump_stack+0xbb/0x10e [ 7839.599302] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7839.600890] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7839.602563] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7839.604094] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7839.605875] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7839.607176] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7839.608678] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7839.610136] ? update_process_times+0x6d/0xc0 [ 7839.611579] ? tick_sched_handle.isra.18+0x27/0x90 [ 7839.612959] ? tick_sched_timer+0x53/0xb0 [ 7839.614146] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7839.615987] ? tick_sched_handle.isra.18+0x90/0x90 [ 7839.617646] ? hrtimer_interrupt+0x128/0x2c0 [ 7839.619281] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7839.620908] ? apic_timer_interrupt+0xf/0x20 [ 7839.622445] [ 7839.623119] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7839.625105] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7839.626952] ? dump_stack+0xf6/0x10e [ 7839.628421] ? do_raw_spin_lock+0x177/0x190 [ 7839.629766] ? _raw_spin_lock+0x21/0x30 [ 7839.630827] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7839.633063] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7839.634878] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7839.636276] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7839.638051] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7839.640226] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7839.642198] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7839.644289] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7839.645719] ? do_wait_intr+0xf0/0xf0 [ 7839.646948] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7839.648974] ? kthread+0x1d1/0x200 [ 7839.650097] ? set_kthread_struct+0x70/0x70 [ 7839.651959] ? ret_from_fork+0x1f/0x30 [ 7848.054185] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 7848.061235] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7848.104276] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7848.114805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7848.123092] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7848.125212] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7848.147304] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7848.152462] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7848.156349] RDX: 00000000000064ff RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7848.160802] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7848.167764] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7848.171429] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7848.181295] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7848.185758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7848.193384] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7848.198607] Call Trace: [ 7848.200536] [ 7848.201529] ? show_regs.cold.9+0x22/0x2f [ 7848.204407] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7848.208677] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7848.212344] ? watchdog+0x40/0x40 [ 7848.213383] ? hrtimer_interrupt+0x128/0x2c0 [ 7848.215538] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7848.218134] ? apic_timer_interrupt+0xf/0x20 [ 7848.221859] [ 7848.222652] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7848.229041] do_raw_spin_lock+0x177/0x190 [ 7848.230496] _raw_spin_lock+0x21/0x30 [ 7848.234340] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7848.239667] ? do_raw_spin_unlock+0x75/0x190 [ 7848.241218] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7848.246377] ? do_wait_intr+0xf0/0xf0 [ 7848.248262] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7848.251004] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7848.253523] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7848.256810] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7848.259402] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7848.261680] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7848.263777] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7848.266053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7848.271855] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7848.274879] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7848.277467] lmv_intent_open+0x338/0x1150 [lmv] [ 7848.278808] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7848.280735] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7848.282699] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7848.284179] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7848.285728] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7848.287471] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7848.289291] ? do_raw_spin_unlock+0x75/0x190 [ 7848.290673] ? _raw_spin_unlock+0x12/0x30 [ 7848.292160] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7848.293694] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7848.295406] ll_atomic_open+0x36a/0xd70 [lustre] [ 7848.297066] lookup_open+0x356/0xb40 [ 7848.298314] do_last+0x313/0xfc0 [ 7848.299285] ? nd_jump_root+0xe5/0x160 [ 7848.300642] ? path_init+0x437/0x520 [ 7848.301902] path_openat+0xf7/0x500 [ 7848.302984] ? do_raw_spin_unlock+0x75/0x190 [ 7848.304419] do_filp_open+0x99/0x140 [ 7848.305542] ? getname_flags+0x6e/0x330 [ 7848.306838] ? __check_object_size+0xff/0x256 [ 7848.308407] ? do_raw_spin_unlock+0x75/0x190 [ 7848.309672] ? _raw_spin_unlock+0x12/0x30 [ 7848.311036] do_sys_openat2+0x2b4/0x410 [ 7848.312306] do_sys_open+0x73/0xa0 [ 7848.313343] __x64_sys_openat+0x24/0x30 [ 7848.314487] do_syscall_64+0xc1/0x440 [ 7848.315728] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7848.317449] RIP: 0033:0x7fdc0afbf332 [ 7848.318727] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7848.324731] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7848.327172] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7848.329282] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7848.331530] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7848.333815] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7848.335814] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7852.002153] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5827s! [ 7852.006187] Showing busy workqueues and worker pools: [ 7852.009567] workqueue events: flags=0x0 [ 7852.011028] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7852.018896] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7852.030480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7852.041530] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7852.053814] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7852.054017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7852.062464] pending: kfree_rcu_monitor [ 7852.065420] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7852.065499] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7852.065507] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7852.065510] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7852.065524] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7852.065528] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7852.065532] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7852.065534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7852.065536] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7852.065538] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7852.065539] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7852.065542] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7852.065544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7852.065546] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7852.065553] Call Trace: [ 7852.065558] [ 7852.065561] ? show_regs.cold.9+0x22/0x2f [ 7852.065568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7852.065573] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7852.065576] ? watchdog+0x40/0x40 [ 7852.065578] ? hrtimer_interrupt+0x128/0x2c0 [ 7852.065581] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7852.065586] ? apic_timer_interrupt+0xf/0x20 [ 7852.065589] [ 7852.065591] ? native_safe_halt+0x2e/0x50 [ 7852.066209] kvm_wait+0xcd/0xe0 [ 7852.066215] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7852.066223] do_raw_spin_lock+0x177/0x190 [ 7852.066228] _raw_spin_lock+0x21/0x30 [ 7852.066232] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7852.066556] ? do_raw_spin_unlock+0x75/0x190 [ 7852.066560] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7852.066734] ? do_wait_intr+0xf0/0xf0 [ 7852.066743] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7852.066881] mdc_close+0x53b/0xd70 [mdc] [ 7852.066912] lmv_close+0x18d/0x490 [lmv] [ 7852.066932] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7852.067155] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7852.067207] ll_md_close+0x274/0x800 [lustre] [ 7852.067282] ll_file_release+0x128/0xc50 [lustre] [ 7852.067437] __fput+0x10d/0x440 [ 7852.067444] ____fput+0x12/0x20 [ 7852.067448] task_work_run+0xc6/0x110 [ 7852.067456] exit_to_usermode_loop+0x1dd/0x1f0 [ 7852.067463] do_syscall_64+0x3d6/0x440 [ 7852.067467] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7852.067474] RIP: 0033:0x7fdc0afbfcd8 [ 7852.067481] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7852.067485] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7852.067490] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7852.067492] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7852.067494] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7852.067497] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7852.067499] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7852.293177] [ 7852.293963] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7852.295963] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7852.303894] workqueue events_long: flags=0x0 [ 7852.306496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7852.310577] pending: lu_objects_flush [obdclass] [ 7852.314892] workqueue events_unbound: flags=0x2 [ 7852.319668] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7852.324116] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7852.330958] workqueue events_power_efficient: flags=0x80 [ 7852.335229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7852.339077] pending: neigh_periodic_work, check_lifetime [ 7852.342260] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7852.346691] pending: do_cache_clean, neigh_periodic_work [ 7852.350151] workqueue mm_percpu_wq: flags=0x8 [ 7852.352467] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7852.359741] pending: lru_add_drain_per_cpu [ 7852.361953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7852.365434] pending: vmstat_update, lru_add_drain_per_cpu [ 7852.369425] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7852.372890] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7852.376642] workqueue writeback: flags=0x4a [ 7852.379371] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7852.383416] pending: wb_workfn, wb_workfn [ 7852.385332] workqueue xprtiod: flags=0xa [ 7852.386728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7852.390452] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7852.394876] workqueue ipv6_addrconf: flags=0x40008 [ 7852.398349] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7852.409240] pending: addrconf_verify_work [ 7852.413899] workqueue knbd0-recv: flags=0x1a [ 7852.416257] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7852.418845] in-flight: 655:recv_work [ 7852.422621] workqueue ptlrpc_pinger: flags=0x2 [ 7852.424646] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7852.427782] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7852.434645] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5827s workers=4 idle: 36396 38 11188 [ 7852.440330] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7842s workers=2 idle: 658 [ 7864.051026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 7864.053375] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7864.068004] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7864.072744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7864.075914] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7864.078162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7864.080218] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7864.089105] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7864.091652] RAX: 00000000000061ac RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7864.097659] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7864.101054] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7864.103970] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7864.106638] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7864.110211] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7864.113669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7864.116628] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7864.120572] Call Trace: [ 7864.123334] [ 7864.124848] ? show_regs.cold.9+0x22/0x2f [ 7864.127823] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7864.131408] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7864.134642] ? watchdog+0x40/0x40 [ 7864.136888] ? hrtimer_interrupt+0x128/0x2c0 [ 7864.138149] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7864.142977] ? apic_timer_interrupt+0xf/0x20 [ 7864.146867] [ 7864.148165] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7864.150130] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7864.151709] do_raw_spin_lock+0x177/0x190 [ 7864.159198] _raw_spin_lock+0x21/0x30 [ 7864.161902] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7864.164870] process_one_work+0x2c8/0x700 [ 7864.167408] worker_thread+0x296/0x6e0 [ 7864.169476] ? rescuer_thread+0x570/0x570 [ 7864.172866] kthread+0x1d1/0x200 [ 7864.174752] ? set_kthread_struct+0x70/0x70 [ 7864.177432] ret_from_fork+0x1f/0x30 [ 7876.052018] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 7876.055079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7876.074028] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7876.078460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7876.081785] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7876.084327] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7876.091467] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7876.095064] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7876.098079] RDX: 0000000000004496 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7876.101263] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7876.104641] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7876.108068] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7876.111362] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7876.117509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7876.120272] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7876.123586] Call Trace: [ 7876.124820] [ 7876.125638] ? show_regs.cold.9+0x22/0x2f [ 7876.127276] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7876.131248] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7876.133474] ? watchdog+0x40/0x40 [ 7876.135138] ? hrtimer_interrupt+0x128/0x2c0 [ 7876.137104] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7876.139468] ? apic_timer_interrupt+0xf/0x20 [ 7876.141348] [ 7876.142937] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7876.145305] do_raw_spin_lock+0x177/0x190 [ 7876.150794] _raw_spin_lock+0x21/0x30 [ 7876.154292] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7876.157404] ? do_raw_spin_unlock+0x75/0x190 [ 7876.161151] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7876.163575] ? do_wait_intr+0xf0/0xf0 [ 7876.165392] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7876.168023] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7876.170359] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7876.176098] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7876.179045] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7876.181487] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7876.183176] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7876.185946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7876.188068] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7876.191005] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7876.195087] lmv_intent_open+0x338/0x1150 [lmv] [ 7876.198128] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7876.200636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7876.203212] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7876.205313] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7876.207284] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7876.209515] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7876.211329] ? do_raw_spin_unlock+0x75/0x190 [ 7876.213485] ? _raw_spin_unlock+0x12/0x30 [ 7876.215373] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7876.217640] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7876.220097] ll_atomic_open+0x36a/0xd70 [lustre] [ 7876.222067] lookup_open+0x356/0xb40 [ 7876.226399] do_last+0x313/0xfc0 [ 7876.227672] ? nd_jump_root+0xe5/0x160 [ 7876.229418] ? path_init+0x437/0x520 [ 7876.231523] path_openat+0xf7/0x500 [ 7876.235437] ? do_raw_spin_unlock+0x75/0x190 [ 7876.239063] do_filp_open+0x99/0x140 [ 7876.241832] ? getname_flags+0x6e/0x330 [ 7876.244883] ? __check_object_size+0xff/0x256 [ 7876.247412] ? do_raw_spin_unlock+0x75/0x190 [ 7876.249384] ? _raw_spin_unlock+0x12/0x30 [ 7876.250955] do_sys_openat2+0x2b4/0x410 [ 7876.253002] do_sys_open+0x73/0xa0 [ 7876.254399] __x64_sys_openat+0x24/0x30 [ 7876.255479] do_syscall_64+0xc1/0x440 [ 7876.258142] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7876.260122] RIP: 0033:0x7fdc0afbf332 [ 7876.263541] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7876.279138] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7876.285959] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7876.289040] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7876.292806] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7876.295406] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7876.297912] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7880.048072] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7880.058773] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7880.065851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7880.066062] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7880.066069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7880.066072] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7880.066085] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7880.066089] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7880.066093] RAX: 00000000000016b0 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7880.066095] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7880.066097] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7880.066099] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7880.066101] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7880.066104] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7880.066107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7880.078630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7880.078709] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7880.078717] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7880.078720] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7880.078778] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7880.078784] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7880.119782] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7880.124616] RAX: 0000000000006940 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7880.128251] Call Trace: [ 7880.130754] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7880.130763] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7880.130765] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7880.130768] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7880.130771] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7880.130774] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7880.130777] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7880.130784] Call Trace: [ 7880.130791] [ 7880.130797] ? show_regs.cold.9+0x22/0x2f [ 7880.130811] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7880.130818] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7880.130822] ? watchdog+0x40/0x40 [ 7880.130826] ? hrtimer_interrupt+0x128/0x2c0 [ 7880.130830] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7880.130838] ? apic_timer_interrupt+0xf/0x20 [ 7880.130906] [ 7880.130911] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7880.130921] do_raw_spin_lock+0x177/0x190 [ 7880.130925] _raw_spin_lock+0x21/0x30 [ 7880.130930] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7880.131269] ? do_raw_spin_unlock+0x75/0x190 [ 7880.131276] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7880.147805] [ 7880.150486] ? do_wait_intr+0xf0/0xf0 [ 7880.154138] ? show_regs.cold.9+0x22/0x2f [ 7880.154152] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7880.154159] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7880.154163] ? watchdog+0x40/0x40 [ 7880.154173] ? hrtimer_interrupt+0x128/0x2c0 [ 7880.154177] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7880.154186] ? apic_timer_interrupt+0xf/0x20 [ 7880.154191] [ 7880.154195] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7880.154201] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7880.154205] ? dump_stack+0xf6/0x10e [ 7880.154210] do_raw_spin_lock+0x177/0x190 [ 7880.154215] _raw_spin_lock+0x21/0x30 [ 7880.154222] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7880.154569] obd_mod_free+0x46/0x140 [ptlrpc] [ 7880.154778] mdc_commit_open+0x91/0xb0 [mdc] [ 7880.154817] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7880.155038] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7880.155254] after_reply+0xe2c/0x1780 [ptlrpc] [ 7880.155459] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7880.155682] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7880.155909] ? do_wait_intr+0xf0/0xf0 [ 7880.155919] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7880.156120] kthread+0x1d1/0x200 [ 7880.156126] ? set_kthread_struct+0x70/0x70 [ 7880.156130] ret_from_fork+0x1f/0x30 [ 7880.325570] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7880.327988] mdc_close+0x53b/0xd70 [mdc] [ 7880.329742] lmv_close+0x18d/0x490 [lmv] [ 7880.331954] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7880.334601] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7880.337195] ll_md_close+0x274/0x800 [lustre] [ 7880.339416] ll_file_release+0x128/0xc50 [lustre] [ 7880.341801] __fput+0x10d/0x440 [ 7880.342884] ____fput+0x12/0x20 [ 7880.343816] task_work_run+0xc6/0x110 [ 7880.345236] exit_to_usermode_loop+0x1dd/0x1f0 [ 7880.347050] do_syscall_64+0x3d6/0x440 [ 7880.348664] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7880.350671] RIP: 0033:0x7fdc0afbfcd8 [ 7880.351685] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7880.359164] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7880.362302] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7880.365224] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7880.368632] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7880.371429] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7880.374276] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7882.719039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5857s! [ 7882.722699] Showing busy workqueues and worker pools: [ 7882.724663] workqueue events: flags=0x0 [ 7882.725763] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7882.727407] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7882.730808] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7882.735317] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7882.739624] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7882.741455] pending: kfree_rcu_monitor [ 7882.742728] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7882.745384] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7882.749969] workqueue events_long: flags=0x0 [ 7882.751583] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7882.753595] pending: lu_objects_flush [obdclass] [ 7882.755334] workqueue events_unbound: flags=0x2 [ 7882.756686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7882.758526] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7882.761675] workqueue events_power_efficient: flags=0x80 [ 7882.763531] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7882.765755] pending: neigh_periodic_work, check_lifetime [ 7882.767684] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7882.770291] pending: do_cache_clean, neigh_periodic_work [ 7882.772244] workqueue mm_percpu_wq: flags=0x8 [ 7882.773982] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7882.776285] pending: lru_add_drain_per_cpu [ 7882.777634] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7882.780068] pending: vmstat_update, lru_add_drain_per_cpu [ 7882.782103] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7882.784537] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7882.787369] workqueue writeback: flags=0x4a [ 7882.788741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7882.791486] pending: wb_workfn, wb_workfn [ 7882.793338] workqueue xprtiod: flags=0xa [ 7882.794991] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7882.797087] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7882.804246] workqueue ipv6_addrconf: flags=0x40008 [ 7882.806541] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7882.808576] pending: addrconf_verify_work [ 7882.809911] workqueue knbd0-recv: flags=0x1a [ 7882.811174] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7882.812900] in-flight: 655:recv_work [ 7882.814023] workqueue ptlrpc_pinger: flags=0x2 [ 7882.815266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7882.816962] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7882.818715] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5857s workers=4 idle: 36396 38 11188 [ 7882.821113] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7873s workers=2 idle: 658 [ 7892.051062] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7892.054989] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7892.081873] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7892.088335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7892.094662] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7892.100149] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7892.101888] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7892.110795] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7892.114006] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7892.116518] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7892.119348] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7892.122469] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7892.124819] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7892.126792] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7892.130345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7892.132631] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7892.135588] Call Trace: [ 7892.136499] [ 7892.137184] ? show_regs.cold.9+0x22/0x2f [ 7892.139518] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7892.141207] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7892.144145] ? watchdog+0x40/0x40 [ 7892.145514] ? hrtimer_interrupt+0x128/0x2c0 [ 7892.148825] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7892.150551] ? apic_timer_interrupt+0xf/0x20 [ 7892.153776] [ 7892.154548] ? native_safe_halt+0x2e/0x50 [ 7892.158067] kvm_wait+0xcd/0xe0 [ 7892.159417] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7892.162170] do_raw_spin_lock+0x177/0x190 [ 7892.163605] _raw_spin_lock+0x21/0x30 [ 7892.167119] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7892.171546] process_one_work+0x2c8/0x700 [ 7892.173127] worker_thread+0x296/0x6e0 [ 7892.176436] ? rescuer_thread+0x570/0x570 [ 7892.180422] kthread+0x1d1/0x200 [ 7892.183960] ? set_kthread_struct+0x70/0x70 [ 7892.185484] ret_from_fork+0x1f/0x30 [ 7904.062691] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 7904.066808] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7904.110659] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7904.117243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7904.135683] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7904.146681] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7904.158644] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7904.163001] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7904.167867] RDX: 0000000000003958 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7904.181838] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7904.190871] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7904.200231] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7904.204939] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7904.209206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7904.212521] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7904.217290] Call Trace: [ 7904.218975] [ 7904.220188] ? show_regs.cold.9+0x22/0x2f [ 7904.223214] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7904.225949] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7904.228592] ? watchdog+0x40/0x40 [ 7904.230138] ? hrtimer_interrupt+0x128/0x2c0 [ 7904.232273] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7904.234816] ? apic_timer_interrupt+0xf/0x20 [ 7904.237152] [ 7904.238101] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7904.265598] do_raw_spin_lock+0x177/0x190 [ 7904.268239] _raw_spin_lock+0x21/0x30 [ 7904.270351] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7904.273949] ? do_raw_spin_unlock+0x75/0x190 [ 7904.276882] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7904.279920] ? do_wait_intr+0xf0/0xf0 [ 7904.285846] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7904.288964] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7904.291890] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7904.295011] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7904.297643] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7904.301470] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7904.304349] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7904.307092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7904.310102] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7904.315006] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7904.318438] lmv_intent_open+0x338/0x1150 [lmv] [ 7904.321601] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7904.324354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7904.328225] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7904.331271] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7904.338101] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7904.348940] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7904.352408] ? do_raw_spin_unlock+0x75/0x190 [ 7904.355841] ? _raw_spin_unlock+0x12/0x30 [ 7904.357698] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7904.363310] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7904.368280] ll_atomic_open+0x36a/0xd70 [lustre] [ 7904.373062] lookup_open+0x356/0xb40 [ 7904.375402] do_last+0x313/0xfc0 [ 7904.379378] ? nd_jump_root+0xe5/0x160 [ 7904.383595] ? path_init+0x437/0x520 [ 7904.386746] path_openat+0xf7/0x500 [ 7904.388933] ? do_raw_spin_unlock+0x75/0x190 [ 7904.392593] do_filp_open+0x99/0x140 [ 7904.395308] ? getname_flags+0x6e/0x330 [ 7904.397907] ? __check_object_size+0xff/0x256 [ 7904.400756] ? do_raw_spin_unlock+0x75/0x190 [ 7904.403195] ? _raw_spin_unlock+0x12/0x30 [ 7904.405691] do_sys_openat2+0x2b4/0x410 [ 7904.407405] do_sys_open+0x73/0xa0 [ 7904.409652] __x64_sys_openat+0x24/0x30 [ 7904.411836] do_syscall_64+0xc1/0x440 [ 7904.413926] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7904.416904] RIP: 0033:0x7fdc0afbf332 [ 7904.418940] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7904.441692] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7904.447408] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7904.451214] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7904.454856] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7904.458660] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7904.462385] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7908.052908] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7908.054064] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 7908.062281] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7908.062365] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7908.062384] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7908.062387] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7908.062401] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7908.062414] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7908.062419] RAX: 0000000000006c84 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7908.062421] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7908.062424] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7908.062426] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7908.062428] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7908.062432] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7908.062434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7908.062436] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7908.062444] Call Trace: [ 7908.062449] [ 7908.062455] ? show_regs.cold.9+0x22/0x2f [ 7908.062466] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7908.062472] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7908.062477] ? watchdog+0x40/0x40 [ 7908.062482] ? hrtimer_interrupt+0x128/0x2c0 [ 7908.062487] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7908.062496] ? apic_timer_interrupt+0xf/0x20 [ 7908.062502] [ 7908.062517] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7908.062523] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7908.062527] ? dump_stack+0xf6/0x10e [ 7908.062532] do_raw_spin_lock+0x177/0x190 [ 7908.062538] _raw_spin_lock+0x21/0x30 [ 7908.062544] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7908.063274] obd_mod_free+0x46/0x140 [ptlrpc] [ 7908.063482] mdc_commit_open+0x91/0xb0 [mdc] [ 7908.063532] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7908.069828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7908.069907] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7908.069915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7908.069918] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7908.069933] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7908.069937] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7908.069942] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7908.069945] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7908.069947] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7908.069949] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7908.069952] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7908.069955] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7908.069958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7908.069961] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7908.069970] Call Trace: [ 7908.069975] [ 7908.069981] ? show_regs.cold.9+0x22/0x2f [ 7908.069990] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7908.070055] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7908.070060] ? watchdog+0x40/0x40 [ 7908.070064] ? hrtimer_interrupt+0x128/0x2c0 [ 7908.070069] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7908.070075] ? apic_timer_interrupt+0xf/0x20 [ 7908.070080] [ 7908.070082] ? native_safe_halt+0x2e/0x50 [ 7908.070092] kvm_wait+0xcd/0xe0 [ 7908.070100] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7908.070109] do_raw_spin_lock+0x177/0x190 [ 7908.070113] _raw_spin_lock+0x21/0x30 [ 7908.070117] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7908.070404] ? do_raw_spin_unlock+0x75/0x190 [ 7908.070420] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7908.070624] ? do_wait_intr+0xf0/0xf0 [ 7908.070632] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7908.126928] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7908.136218] mdc_close+0x53b/0xd70 [mdc] [ 7908.136279] lmv_close+0x18d/0x490 [lmv] [ 7908.136315] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7908.136459] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7908.136538] ll_md_close+0x274/0x800 [lustre] [ 7908.136627] ll_file_release+0x128/0xc50 [lustre] [ 7908.137250] __fput+0x10d/0x440 [ 7908.137261] ____fput+0x12/0x20 [ 7908.137265] task_work_run+0xc6/0x110 [ 7908.137272] exit_to_usermode_loop+0x1dd/0x1f0 [ 7908.137281] do_syscall_64+0x3d6/0x440 [ 7908.137286] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7908.149582] after_reply+0xe2c/0x1780 [ptlrpc] [ 7908.149934] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7908.150183] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7908.150384] ? do_wait_intr+0xf0/0xf0 [ 7908.150396] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7908.150608] kthread+0x1d1/0x200 [ 7908.150615] ? set_kthread_struct+0x70/0x70 [ 7908.150619] ret_from_fork+0x1f/0x30 [ 7908.394345] RIP: 0033:0x7fdc0afbfcd8 [ 7908.396880] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7908.406380] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7908.408871] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7908.411286] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7908.414203] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7908.417141] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7908.419914] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7912.929464] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5888s! [ 7912.937605] Showing busy workqueues and worker pools: [ 7912.945620] workqueue events: flags=0x0 [ 7912.948191] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7912.953228] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7912.958543] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7912.963168] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7912.970035] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7912.974728] pending: kfree_rcu_monitor [ 7912.976349] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7912.980290] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7912.986869] workqueue events_long: flags=0x0 [ 7912.988065] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7912.998206] pending: lu_objects_flush [obdclass] [ 7913.004011] workqueue events_unbound: flags=0x2 [ 7913.008355] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7913.013831] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7913.022285] workqueue events_power_efficient: flags=0x80 [ 7913.027196] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7913.032429] pending: neigh_periodic_work, check_lifetime [ 7913.037161] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7913.043991] pending: do_cache_clean, neigh_periodic_work [ 7913.048963] workqueue mm_percpu_wq: flags=0x8 [ 7913.053142] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7913.057474] pending: lru_add_drain_per_cpu [ 7913.059457] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7913.063597] pending: vmstat_update, lru_add_drain_per_cpu [ 7913.069930] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7913.074369] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7913.077592] workqueue writeback: flags=0x4a [ 7913.079311] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7913.082086] pending: wb_workfn, wb_workfn [ 7913.084100] workqueue xprtiod: flags=0xa [ 7913.086177] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7913.091757] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7913.095160] workqueue ipv6_addrconf: flags=0x40008 [ 7913.097033] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7913.100041] pending: addrconf_verify_work [ 7913.106594] workqueue knbd0-recv: flags=0x1a [ 7913.110460] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7913.114761] in-flight: 655:recv_work [ 7913.121187] workqueue ptlrpc_pinger: flags=0x2 [ 7913.122634] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7913.128085] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7913.136242] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5888s workers=4 idle: 36396 38 11188 [ 7913.141765] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7903s workers=2 idle: 658 [ 7920.051152] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7920.055053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7920.076230] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7920.081412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7920.085908] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7920.089627] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7920.092658] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7920.102429] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7920.106545] RAX: 0000000000002db7 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7920.110255] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7920.113741] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7920.117862] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7920.122407] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7920.125774] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7920.130535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7920.132637] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7920.136133] Call Trace: [ 7920.137160] [ 7920.137964] ? show_regs.cold.9+0x22/0x2f [ 7920.139844] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7920.141413] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7920.143531] ? watchdog+0x40/0x40 [ 7920.144969] ? hrtimer_interrupt+0x128/0x2c0 [ 7920.146917] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7920.149380] ? apic_timer_interrupt+0xf/0x20 [ 7920.151417] [ 7920.152299] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7920.154722] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7920.157318] do_raw_spin_lock+0x177/0x190 [ 7920.159387] _raw_spin_lock+0x21/0x30 [ 7920.161209] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7920.164466] process_one_work+0x2c8/0x700 [ 7920.165990] worker_thread+0x296/0x6e0 [ 7920.167372] ? rescuer_thread+0x570/0x570 [ 7920.168910] kthread+0x1d1/0x200 [ 7920.170023] ? set_kthread_struct+0x70/0x70 [ 7920.171547] ret_from_fork+0x1f/0x30 [ 7932.052985] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 7932.059895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7932.126592] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7932.142908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7932.151506] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7932.155303] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7932.171130] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7932.175409] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7932.179175] RDX: 0000000000007e0d RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7932.182252] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7932.186145] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7932.190203] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7932.193394] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7932.197552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7932.205344] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7932.212362] Call Trace: [ 7932.215991] [ 7932.217745] ? show_regs.cold.9+0x22/0x2f [ 7932.220786] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7932.224840] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7932.226309] ? watchdog+0x40/0x40 [ 7932.227207] ? hrtimer_interrupt+0x128/0x2c0 [ 7932.230261] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7932.233606] ? apic_timer_interrupt+0xf/0x20 [ 7932.239093] [ 7932.240807] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7932.244333] do_raw_spin_lock+0x177/0x190 [ 7932.248455] _raw_spin_lock+0x21/0x30 [ 7932.254137] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7932.259399] ? do_raw_spin_unlock+0x75/0x190 [ 7932.262373] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7932.270032] ? do_wait_intr+0xf0/0xf0 [ 7932.271140] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7932.272972] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7932.282544] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7932.293325] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7932.302896] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7932.304465] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7932.309527] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7932.312999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7932.315482] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7932.319002] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7932.323882] lmv_intent_open+0x338/0x1150 [lmv] [ 7932.325278] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7932.333267] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7932.339265] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7932.344218] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7932.345929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7932.351770] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7932.358504] ? do_raw_spin_unlock+0x75/0x190 [ 7932.360689] ? _raw_spin_unlock+0x12/0x30 [ 7932.362211] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7932.365915] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7932.370022] ll_atomic_open+0x36a/0xd70 [lustre] [ 7932.371805] lookup_open+0x356/0xb40 [ 7932.374266] do_last+0x313/0xfc0 [ 7932.375857] ? nd_jump_root+0xe5/0x160 [ 7932.383270] ? path_init+0x437/0x520 [ 7932.384556] path_openat+0xf7/0x500 [ 7932.388089] ? do_raw_spin_unlock+0x75/0x190 [ 7932.393887] do_filp_open+0x99/0x140 [ 7932.397243] ? getname_flags+0x6e/0x330 [ 7932.400767] ? __check_object_size+0xff/0x256 [ 7932.403062] ? do_raw_spin_unlock+0x75/0x190 [ 7932.406721] ? _raw_spin_unlock+0x12/0x30 [ 7932.407923] do_sys_openat2+0x2b4/0x410 [ 7932.409083] do_sys_open+0x73/0xa0 [ 7932.416454] __x64_sys_openat+0x24/0x30 [ 7932.418223] do_syscall_64+0xc1/0x440 [ 7932.420407] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7932.426755] RIP: 0033:0x7fdc0afbf332 [ 7932.429125] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7932.453228] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7932.458776] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7932.462490] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7932.472242] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7932.475360] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7932.480715] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7936.054041] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 7936.064579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7936.105861] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7936.114048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7936.118940] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7936.121133] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7936.133153] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7936.138592] RAX: 0000000000000519 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7936.146936] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7936.152044] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7936.155039] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7936.159412] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7936.163046] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7936.175287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7936.178110] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7936.187815] Call Trace: [ 7936.189114] [ 7936.193231] ? show_regs.cold.9+0x22/0x2f [ 7936.197415] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7936.200678] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7936.202859] ? watchdog+0x40/0x40 [ 7936.206952] ? hrtimer_interrupt+0x128/0x2c0 [ 7936.211544] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7936.213135] ? apic_timer_interrupt+0xf/0x20 [ 7936.227871] [ 7936.228528] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7936.231452] do_raw_spin_lock+0x177/0x190 [ 7936.236222] _raw_spin_lock+0x21/0x30 [ 7936.237630] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7936.242014] ? do_raw_spin_unlock+0x75/0x190 [ 7936.244034] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7936.247781] ? do_wait_intr+0xf0/0xf0 [ 7936.249201] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7936.252062] mdc_close+0x53b/0xd70 [mdc] [ 7936.255191] lmv_close+0x18d/0x490 [lmv] [ 7936.257492] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7936.266926] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7936.273007] ll_md_close+0x274/0x800 [lustre] [ 7936.276150] ll_file_release+0x128/0xc50 [lustre] [ 7936.279094] __fput+0x10d/0x440 [ 7936.280835] ____fput+0x12/0x20 [ 7936.282182] task_work_run+0xc6/0x110 [ 7936.284074] exit_to_usermode_loop+0x1dd/0x1f0 [ 7936.286267] do_syscall_64+0x3d6/0x440 [ 7936.291825] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7936.294468] RIP: 0033:0x7fdc0afbfcd8 [ 7936.296064] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7936.320364] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7936.328185] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7936.336998] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7936.344524] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7936.348423] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7936.351951] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7940.048877] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7940.054679] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7940.114406] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7940.129333] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7940.139613] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7940.141291] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7940.160592] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7940.164101] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7940.167164] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7940.171057] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7940.174936] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7940.178628] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7940.181736] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7940.184412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7940.187090] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7940.190309] Call Trace: [ 7940.191234] [ 7940.191987] ? show_regs.cold.9+0x22/0x2f [ 7940.193986] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7940.195686] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7940.197664] ? watchdog+0x40/0x40 [ 7940.199173] ? hrtimer_interrupt+0x128/0x2c0 [ 7940.201673] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7940.204009] ? apic_timer_interrupt+0xf/0x20 [ 7940.206048] [ 7940.207072] ? native_safe_halt+0x2e/0x50 [ 7940.209079] kvm_wait+0xcd/0xe0 [ 7940.210432] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7940.213166] ? dump_stack+0xf6/0x10e [ 7940.215136] do_raw_spin_lock+0x177/0x190 [ 7940.216713] _raw_spin_lock+0x21/0x30 [ 7940.218452] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7940.221505] obd_mod_free+0x46/0x140 [ptlrpc] [ 7940.223739] mdc_commit_open+0x91/0xb0 [mdc] [ 7940.227130] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7940.229774] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7940.232499] after_reply+0xe2c/0x1780 [ptlrpc] [ 7940.235514] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7940.238270] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7940.240460] ? do_wait_intr+0xf0/0xf0 [ 7940.242619] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7940.245403] kthread+0x1d1/0x200 [ 7940.247050] ? set_kthread_struct+0x70/0x70 [ 7940.248995] ret_from_fork+0x1f/0x30 [ 7943.647072] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5918s! [ 7943.653997] Showing busy workqueues and worker pools: [ 7943.660457] workqueue events: flags=0x0 [ 7943.663954] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7943.688193] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7943.705974] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7943.723350] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7943.735704] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7943.748023] pending: kfree_rcu_monitor [ 7943.757139] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7943.761008] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7943.782024] workqueue events_long: flags=0x0 [ 7943.789214] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7943.798884] pending: lu_objects_flush [obdclass] [ 7943.810211] workqueue events_unbound: flags=0x2 [ 7943.822494] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7943.835443] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7943.845362] workqueue events_power_efficient: flags=0x80 [ 7943.850211] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7943.865920] pending: neigh_periodic_work, check_lifetime [ 7943.878041] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7943.887420] pending: do_cache_clean, neigh_periodic_work [ 7943.902289] workqueue mm_percpu_wq: flags=0x8 [ 7943.910694] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7943.921079] pending: lru_add_drain_per_cpu [ 7943.922686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7943.934801] pending: vmstat_update, lru_add_drain_per_cpu [ 7943.941882] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7943.946147] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7943.961317] workqueue writeback: flags=0x4a [ 7943.966415] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7943.978430] pending: wb_workfn, wb_workfn [ 7943.986570] workqueue xprtiod: flags=0xa [ 7943.995264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7944.002716] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7944.020896] workqueue ipv6_addrconf: flags=0x40008 [ 7944.022542] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7944.037065] pending: addrconf_verify_work [ 7944.040964] workqueue knbd0-recv: flags=0x1a [ 7944.045865] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7944.056436] in-flight: 655:recv_work [ 7944.058014] workqueue ptlrpc_pinger: flags=0x2 [ 7944.067257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7944.084180] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7944.089490] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5919s workers=4 idle: 36396 38 11188 [ 7944.101970] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7934s workers=2 idle: 658 [ 7948.051497] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7948.072753] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7948.130451] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7948.145954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7948.154173] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7948.168531] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7948.172402] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7948.186186] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7948.193763] RAX: 0000000000007e09 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7948.202240] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7948.212090] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7948.221071] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7948.227637] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7948.236125] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7948.241504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7948.244719] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7948.257971] Call Trace: [ 7948.260134] [ 7948.261609] ? show_regs.cold.9+0x22/0x2f [ 7948.268141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7948.275499] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7948.281207] ? watchdog+0x40/0x40 [ 7948.283968] ? hrtimer_interrupt+0x128/0x2c0 [ 7948.286940] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7948.290196] ? apic_timer_interrupt+0xf/0x20 [ 7948.293231] [ 7948.295255] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7948.302546] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7948.305685] do_raw_spin_lock+0x177/0x190 [ 7948.308088] _raw_spin_lock+0x21/0x30 [ 7948.312140] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7948.316899] process_one_work+0x2c8/0x700 [ 7948.318407] worker_thread+0x296/0x6e0 [ 7948.331615] ? rescuer_thread+0x570/0x570 [ 7948.343181] kthread+0x1d1/0x200 [ 7948.347064] ? set_kthread_struct+0x70/0x70 [ 7948.351356] ret_from_fork+0x1f/0x30 [ 7960.052030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7960.055547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7960.095805] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7960.105465] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7960.111594] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7960.115056] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7960.127216] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7960.130376] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7960.134403] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7960.137444] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7960.140471] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7960.144576] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7960.147826] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7960.152798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7960.155337] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7960.157814] Call Trace: [ 7960.158600] [ 7960.159154] ? show_regs.cold.9+0x22/0x2f [ 7960.161721] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7960.164081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7960.167479] ? watchdog+0x40/0x40 [ 7960.169453] ? hrtimer_interrupt+0x128/0x2c0 [ 7960.172360] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7960.174710] ? apic_timer_interrupt+0xf/0x20 [ 7960.176478] [ 7960.177337] ? native_safe_halt+0x2e/0x50 [ 7960.179099] kvm_wait+0xcd/0xe0 [ 7960.180288] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7960.183718] do_raw_spin_lock+0x177/0x190 [ 7960.185524] _raw_spin_lock+0x21/0x30 [ 7960.187052] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7960.189357] ? do_raw_spin_unlock+0x75/0x190 [ 7960.190949] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7960.195290] ? do_wait_intr+0xf0/0xf0 [ 7960.197043] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7960.201088] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7960.203847] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7960.208224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7960.210937] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7960.215581] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7960.219227] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7960.221484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7960.224600] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7960.227605] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7960.232034] lmv_intent_open+0x338/0x1150 [lmv] [ 7960.233550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7960.236401] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7960.238401] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7960.240289] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7960.242011] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7960.244610] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7960.248132] ? do_raw_spin_unlock+0x75/0x190 [ 7960.250036] ? _raw_spin_unlock+0x12/0x30 [ 7960.252252] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7960.254633] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7960.259044] ll_atomic_open+0x36a/0xd70 [lustre] [ 7960.261976] lookup_open+0x356/0xb40 [ 7960.265541] do_last+0x313/0xfc0 [ 7960.267222] ? nd_jump_root+0xe5/0x160 [ 7960.269787] ? path_init+0x437/0x520 [ 7960.274142] path_openat+0xf7/0x500 [ 7960.278757] ? do_raw_spin_unlock+0x75/0x190 [ 7960.281826] do_filp_open+0x99/0x140 [ 7960.283459] ? getname_flags+0x6e/0x330 [ 7960.286660] ? __check_object_size+0xff/0x256 [ 7960.288869] ? do_raw_spin_unlock+0x75/0x190 [ 7960.291697] ? _raw_spin_unlock+0x12/0x30 [ 7960.294196] do_sys_openat2+0x2b4/0x410 [ 7960.297878] do_sys_open+0x73/0xa0 [ 7960.301938] __x64_sys_openat+0x24/0x30 [ 7960.304075] do_syscall_64+0xc1/0x440 [ 7960.306021] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7960.308588] RIP: 0033:0x7fdc0afbf332 [ 7960.310382] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7960.318673] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7960.323991] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7960.329951] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7960.334751] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7960.343285] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7960.357091] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7964.054089] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 7964.067724] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7964.113621] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7964.118625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7964.123403] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7964.125576] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7964.141664] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7964.149102] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7964.160474] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7964.164268] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7964.169880] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7964.172863] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7964.177545] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7964.185917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7964.193604] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7964.200946] Call Trace: [ 7964.202979] [ 7964.208091] ? show_regs.cold.9+0x22/0x2f [ 7964.212260] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7964.215566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7964.218094] ? watchdog+0x40/0x40 [ 7964.219905] ? hrtimer_interrupt+0x128/0x2c0 [ 7964.227504] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7964.234135] ? apic_timer_interrupt+0xf/0x20 [ 7964.237910] [ 7964.238922] ? native_safe_halt+0x2e/0x50 [ 7964.241912] kvm_wait+0xcd/0xe0 [ 7964.243520] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7964.248052] do_raw_spin_lock+0x177/0x190 [ 7964.252069] _raw_spin_lock+0x21/0x30 [ 7964.259386] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7964.265988] ? do_raw_spin_unlock+0x75/0x190 [ 7964.270161] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7964.281780] ? do_wait_intr+0xf0/0xf0 [ 7964.292838] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7964.299851] mdc_close+0x53b/0xd70 [mdc] [ 7964.304249] lmv_close+0x18d/0x490 [lmv] [ 7964.306631] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7964.316959] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7964.318754] ll_md_close+0x274/0x800 [lustre] [ 7964.335851] ll_file_release+0x128/0xc50 [lustre] [ 7964.337597] __fput+0x10d/0x440 [ 7964.342634] ____fput+0x12/0x20 [ 7964.345400] task_work_run+0xc6/0x110 [ 7964.349412] exit_to_usermode_loop+0x1dd/0x1f0 [ 7964.356891] do_syscall_64+0x3d6/0x440 [ 7964.360508] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7964.363448] RIP: 0033:0x7fdc0afbfcd8 [ 7964.370496] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7964.392576] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7964.401113] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7964.404935] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7964.413967] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7964.422078] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7964.428553] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7972.048170] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 7972.050635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7972.067352] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7972.070858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7972.074044] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7972.075333] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7972.082140] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7972.084726] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7972.087320] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 7972.090151] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 7972.092524] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 7972.095108] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 7972.097761] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 7972.100590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7972.102476] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 7972.105297] Call Trace: [ 7972.106223] [ 7972.106987] ? show_regs.cold.9+0x22/0x2f [ 7972.108294] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7972.110276] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7972.111953] ? watchdog+0x40/0x40 [ 7972.113399] ? hrtimer_interrupt+0x128/0x2c0 [ 7972.114632] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7972.116701] ? apic_timer_interrupt+0xf/0x20 [ 7972.118414] [ 7972.119306] ? native_safe_halt+0x2e/0x50 [ 7972.120798] kvm_wait+0xcd/0xe0 [ 7972.121692] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7972.123386] ? dump_stack+0xf6/0x10e [ 7972.124805] do_raw_spin_lock+0x177/0x190 [ 7972.126155] _raw_spin_lock+0x21/0x30 [ 7972.127721] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7972.129981] obd_mod_free+0x46/0x140 [ptlrpc] [ 7972.131912] mdc_commit_open+0x91/0xb0 [mdc] [ 7972.133581] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7972.135699] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7972.138331] after_reply+0xe2c/0x1780 [ptlrpc] [ 7972.140336] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7972.142507] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7972.144764] ? do_wait_intr+0xf0/0xf0 [ 7972.146110] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7972.148028] kthread+0x1d1/0x200 [ 7972.149032] ? set_kthread_struct+0x70/0x70 [ 7972.150439] ret_from_fork+0x1f/0x30 [ 7974.368094] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5949s! [ 7974.373710] Showing busy workqueues and worker pools: [ 7974.375775] workqueue events: flags=0x0 [ 7974.377242] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7974.380057] pending: cpu_latency_work [ptlrpc], push_to_pool [ 7974.382721] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7974.385499] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 7974.389718] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7974.392404] pending: kfree_rcu_monitor [ 7974.393821] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7974.396369] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 7974.401436] workqueue events_long: flags=0x0 [ 7974.403081] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7974.405559] pending: lu_objects_flush [obdclass] [ 7974.407675] workqueue events_unbound: flags=0x2 [ 7974.409491] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7974.411769] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7974.415975] workqueue events_power_efficient: flags=0x80 [ 7974.417940] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7974.420802] pending: neigh_periodic_work, check_lifetime [ 7974.422698] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7974.425112] pending: do_cache_clean, neigh_periodic_work [ 7974.427108] workqueue mm_percpu_wq: flags=0x8 [ 7974.428524] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7974.431102] pending: lru_add_drain_per_cpu [ 7974.432784] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7974.435393] pending: vmstat_update, lru_add_drain_per_cpu [ 7974.437488] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7974.439938] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7974.442436] workqueue writeback: flags=0x4a [ 7974.443974] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7974.446397] pending: wb_workfn, wb_workfn [ 7974.447916] workqueue xprtiod: flags=0xa [ 7974.449656] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7974.452123] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7974.455376] workqueue ipv6_addrconf: flags=0x40008 [ 7974.457232] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7974.459656] pending: addrconf_verify_work [ 7974.461667] workqueue knbd0-recv: flags=0x1a [ 7974.463458] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7974.465954] in-flight: 655:recv_work [ 7974.467643] workqueue ptlrpc_pinger: flags=0x2 [ 7974.469556] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7974.472015] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 7974.475050] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5949s workers=4 idle: 36396 38 11188 [ 7974.478713] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7965s workers=2 idle: 658 [ 7976.051141] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 7976.057004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7976.094865] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7976.103285] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7976.108503] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7976.112542] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7976.117537] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7976.132266] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7976.135042] RAX: 0000000000007f8d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7976.142397] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 7976.145893] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7976.149264] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 7976.151789] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 7976.168026] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 7976.170136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7976.171480] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 7976.173741] Call Trace: [ 7976.174502] [ 7976.175154] ? show_regs.cold.9+0x22/0x2f [ 7976.176244] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7976.177753] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7976.179476] ? watchdog+0x40/0x40 [ 7976.180759] ? hrtimer_interrupt+0x128/0x2c0 [ 7976.182424] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7976.184275] ? apic_timer_interrupt+0xf/0x20 [ 7976.186107] [ 7976.186682] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7976.188438] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7976.191551] do_raw_spin_lock+0x177/0x190 [ 7976.193269] _raw_spin_lock+0x21/0x30 [ 7976.198422] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7976.200064] process_one_work+0x2c8/0x700 [ 7976.201090] worker_thread+0x296/0x6e0 [ 7976.201982] ? rescuer_thread+0x570/0x570 [ 7976.203043] kthread+0x1d1/0x200 [ 7976.203727] ? set_kthread_struct+0x70/0x70 [ 7976.204763] ret_from_fork+0x1f/0x30 [ 7988.052059] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 7988.055018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7988.084628] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7988.092553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7988.097031] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7988.099753] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7988.108751] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7988.113409] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 7988.117326] RDX: 0000000000007408 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 7988.120130] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 7988.125922] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 7988.135574] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7988.140498] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 7988.144187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7988.146575] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 7988.150136] Call Trace: [ 7988.151376] [ 7988.153106] ? show_regs.cold.9+0x22/0x2f [ 7988.155163] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7988.158059] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7988.161615] ? watchdog+0x40/0x40 [ 7988.166885] ? hrtimer_interrupt+0x128/0x2c0 [ 7988.167976] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7988.172358] ? apic_timer_interrupt+0xf/0x20 [ 7988.177378] [ 7988.179612] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7988.181894] do_raw_spin_lock+0x177/0x190 [ 7988.185332] _raw_spin_lock+0x21/0x30 [ 7988.188897] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7988.192758] ? do_raw_spin_unlock+0x75/0x190 [ 7988.197936] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7988.203267] ? do_wait_intr+0xf0/0xf0 [ 7988.213326] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7988.217362] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7988.221918] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7988.224501] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7988.226365] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7988.229375] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7988.234742] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7988.238420] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7988.245523] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7988.249199] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7988.251450] lmv_intent_open+0x338/0x1150 [lmv] [ 7988.261236] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7988.263336] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7988.273440] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7988.282164] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7988.286486] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7988.290718] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7988.295444] ? do_raw_spin_unlock+0x75/0x190 [ 7988.297147] ? _raw_spin_unlock+0x12/0x30 [ 7988.299395] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7988.303096] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7988.307308] ll_atomic_open+0x36a/0xd70 [lustre] [ 7988.309460] lookup_open+0x356/0xb40 [ 7988.312530] do_last+0x313/0xfc0 [ 7988.313838] ? nd_jump_root+0xe5/0x160 [ 7988.318373] ? path_init+0x437/0x520 [ 7988.320464] path_openat+0xf7/0x500 [ 7988.322513] ? do_raw_spin_unlock+0x75/0x190 [ 7988.324296] do_filp_open+0x99/0x140 [ 7988.325569] ? getname_flags+0x6e/0x330 [ 7988.327048] ? __check_object_size+0xff/0x256 [ 7988.328554] ? do_raw_spin_unlock+0x75/0x190 [ 7988.330152] ? _raw_spin_unlock+0x12/0x30 [ 7988.331718] do_sys_openat2+0x2b4/0x410 [ 7988.333113] do_sys_open+0x73/0xa0 [ 7988.334397] __x64_sys_openat+0x24/0x30 [ 7988.336052] do_syscall_64+0xc1/0x440 [ 7988.337342] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7988.338928] RIP: 0033:0x7fdc0afbf332 [ 7988.340221] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7988.349934] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7988.355488] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 7988.360285] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 7988.363254] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 7988.366121] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 7988.368702] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 7992.055565] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 7992.066977] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7992.113080] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7992.116891] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7992.121047] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7992.123231] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7992.135281] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7992.139192] RAX: 0000000000005b8b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 7992.143931] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 7992.147587] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7992.151413] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 7992.156250] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 7992.162112] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 7992.170989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7992.177301] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 7992.180981] Call Trace: [ 7992.182180] [ 7992.183418] ? show_regs.cold.9+0x22/0x2f [ 7992.187922] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7992.192266] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7992.197036] ? watchdog+0x40/0x40 [ 7992.199241] ? hrtimer_interrupt+0x128/0x2c0 [ 7992.200470] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7992.204909] ? apic_timer_interrupt+0xf/0x20 [ 7992.208542] [ 7992.209951] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7992.213355] do_raw_spin_lock+0x177/0x190 [ 7992.215489] _raw_spin_lock+0x21/0x30 [ 7992.218338] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7992.222305] ? do_raw_spin_unlock+0x75/0x190 [ 7992.227079] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7992.232886] ? do_wait_intr+0xf0/0xf0 [ 7992.234333] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7992.237428] mdc_close+0x53b/0xd70 [mdc] [ 7992.239358] lmv_close+0x18d/0x490 [lmv] [ 7992.241789] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7992.245262] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7992.247284] ll_md_close+0x274/0x800 [lustre] [ 7992.249225] ll_file_release+0x128/0xc50 [lustre] [ 7992.251868] __fput+0x10d/0x440 [ 7992.253075] ____fput+0x12/0x20 [ 7992.254293] task_work_run+0xc6/0x110 [ 7992.255841] exit_to_usermode_loop+0x1dd/0x1f0 [ 7992.256975] do_syscall_64+0x3d6/0x440 [ 7992.258152] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7992.259740] RIP: 0033:0x7fdc0afbfcd8 [ 7992.263011] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7992.273022] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7992.276980] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 7992.283020] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 7992.286193] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 7992.290176] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 7992.299841] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8000.048849] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8000.052838] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8000.099855] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8000.108740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8000.115176] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8000.118323] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8000.143872] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8000.154214] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8000.159100] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8000.162791] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8000.166166] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8000.169161] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8000.172198] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8000.176642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8000.179722] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8000.183919] Call Trace: [ 8000.185675] [ 8000.187197] ? show_regs.cold.9+0x22/0x2f [ 8000.188875] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8000.191609] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8000.193537] ? watchdog+0x40/0x40 [ 8000.194469] ? hrtimer_interrupt+0x128/0x2c0 [ 8000.196121] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8000.198138] ? apic_timer_interrupt+0xf/0x20 [ 8000.201034] [ 8000.201837] ? native_safe_halt+0x2e/0x50 [ 8000.203129] kvm_wait+0xcd/0xe0 [ 8000.204707] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8000.207941] ? dump_stack+0xf6/0x10e [ 8000.209923] do_raw_spin_lock+0x177/0x190 [ 8000.211356] _raw_spin_lock+0x21/0x30 [ 8000.213437] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8000.215446] obd_mod_free+0x46/0x140 [ptlrpc] [ 8000.219169] mdc_commit_open+0x91/0xb0 [mdc] [ 8000.221073] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8000.227209] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8000.229782] after_reply+0xe2c/0x1780 [ptlrpc] [ 8000.236259] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8000.240341] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8000.246012] ? do_wait_intr+0xf0/0xf0 [ 8000.247485] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8000.251088] kthread+0x1d1/0x200 [ 8000.252634] ? set_kthread_struct+0x70/0x70 [ 8000.255880] ret_from_fork+0x1f/0x30 [ 8004.055705] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8004.065771] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8004.115941] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8004.128198] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8004.131570] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8004.134746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8004.141490] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8004.159014] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8004.164371] RAX: 0000000000001d86 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8004.169177] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8004.175297] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8004.183096] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8004.188175] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8004.193945] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8004.200002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8004.204912] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8004.209014] Call Trace: [ 8004.210275] [ 8004.214136] ? show_regs.cold.9+0x22/0x2f [ 8004.216781] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8004.219092] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8004.223000] ? watchdog+0x40/0x40 [ 8004.225170] ? hrtimer_interrupt+0x128/0x2c0 [ 8004.230651] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8004.232305] ? apic_timer_interrupt+0xf/0x20 [ 8004.234259] [ 8004.235218] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8004.238499] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8004.241781] do_raw_spin_lock+0x177/0x190 [ 8004.248142] _raw_spin_lock+0x21/0x30 [ 8004.250214] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8004.256837] process_one_work+0x2c8/0x700 [ 8004.259560] worker_thread+0x296/0x6e0 [ 8004.261018] ? rescuer_thread+0x570/0x570 [ 8004.263637] kthread+0x1d1/0x200 [ 8004.266299] ? set_kthread_struct+0x70/0x70 [ 8004.268032] ret_from_fork+0x1f/0x30 [ 8004.575452] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5979s! [ 8004.580116] Showing busy workqueues and worker pools: [ 8004.583984] workqueue events: flags=0x0 [ 8004.587170] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8004.593796] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8004.597161] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8004.600763] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8004.605989] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8004.610254] pending: kfree_rcu_monitor [ 8004.613327] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8004.620184] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8004.626610] workqueue events_long: flags=0x0 [ 8004.631140] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8004.640360] pending: lu_objects_flush [obdclass] [ 8004.645960] workqueue events_unbound: flags=0x2 [ 8004.651977] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8004.655244] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8004.664617] workqueue events_power_efficient: flags=0x80 [ 8004.666014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8004.672237] pending: neigh_periodic_work, check_lifetime [ 8004.676619] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8004.683108] pending: do_cache_clean, neigh_periodic_work [ 8004.692984] workqueue mm_percpu_wq: flags=0x8 [ 8004.695624] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8004.698311] pending: lru_add_drain_per_cpu [ 8004.703010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8004.707240] pending: vmstat_update, lru_add_drain_per_cpu [ 8004.711104] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8004.715828] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8004.720606] workqueue writeback: flags=0x4a [ 8004.722809] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8004.726001] pending: wb_workfn, wb_workfn [ 8004.728587] workqueue xprtiod: flags=0xa [ 8004.730661] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8004.734123] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8004.738111] workqueue ipv6_addrconf: flags=0x40008 [ 8004.740502] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8004.743637] pending: addrconf_verify_work [ 8004.746143] workqueue knbd0-recv: flags=0x1a [ 8004.748057] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8004.751497] in-flight: 655:recv_work [ 8004.753741] workqueue ptlrpc_pinger: flags=0x2 [ 8004.755927] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8004.758790] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8004.762154] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5979s workers=4 idle: 36396 38 11188 [ 8004.766659] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7995s workers=2 idle: 658 [ 8016.052171] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8016.055036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8016.075627] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8016.081118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8016.085161] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8016.087254] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8016.095129] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8016.098527] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8016.102027] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8016.105481] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8016.108126] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8016.111008] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8016.114469] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8016.117768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8016.120099] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8016.122923] Call Trace: [ 8016.123968] [ 8016.125067] ? show_regs.cold.9+0x22/0x2f [ 8016.126649] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8016.128559] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8016.130604] ? watchdog+0x40/0x40 [ 8016.131927] ? hrtimer_interrupt+0x128/0x2c0 [ 8016.133769] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8016.135649] ? apic_timer_interrupt+0xf/0x20 [ 8016.137483] [ 8016.138132] ? native_safe_halt+0x2e/0x50 [ 8016.139803] kvm_wait+0xcd/0xe0 [ 8016.140864] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8016.142828] do_raw_spin_lock+0x177/0x190 [ 8016.144043] _raw_spin_lock+0x21/0x30 [ 8016.145509] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8016.149548] ? do_raw_spin_unlock+0x75/0x190 [ 8016.151452] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8016.153873] ? do_wait_intr+0xf0/0xf0 [ 8016.155297] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8016.157645] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8016.160129] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8016.163099] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8016.165915] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8016.168707] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8016.171156] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8016.175019] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8016.177740] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8016.182145] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8016.185864] lmv_intent_open+0x338/0x1150 [lmv] [ 8016.187895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8016.190598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8016.192838] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8016.194928] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8016.196776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8016.198950] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8016.200738] ? do_raw_spin_unlock+0x75/0x190 [ 8016.202432] ? _raw_spin_unlock+0x12/0x30 [ 8016.204234] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8016.206282] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8016.207994] ll_atomic_open+0x36a/0xd70 [lustre] [ 8016.209738] lookup_open+0x356/0xb40 [ 8016.211021] do_last+0x313/0xfc0 [ 8016.212439] ? nd_jump_root+0xe5/0x160 [ 8016.213913] ? path_init+0x437/0x520 [ 8016.215409] path_openat+0xf7/0x500 [ 8016.216961] ? do_raw_spin_unlock+0x75/0x190 [ 8016.218603] do_filp_open+0x99/0x140 [ 8016.220105] ? getname_flags+0x6e/0x330 [ 8016.221703] ? __check_object_size+0xff/0x256 [ 8016.223367] ? do_raw_spin_unlock+0x75/0x190 [ 8016.224996] ? _raw_spin_unlock+0x12/0x30 [ 8016.227065] do_sys_openat2+0x2b4/0x410 [ 8016.228824] do_sys_open+0x73/0xa0 [ 8016.230129] __x64_sys_openat+0x24/0x30 [ 8016.231684] do_syscall_64+0xc1/0x440 [ 8016.233332] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8016.234986] RIP: 0033:0x7fdc0afbf332 [ 8016.236410] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8016.242953] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8016.245723] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8016.248111] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8016.250668] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8016.252977] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8016.255456] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8019.490042] rcu: INFO: rcu_sched self-detected stall on CPU [ 8019.495313] rcu: 0-...!: (3752954 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 8019.501843] (t=6000147 jiffies g=270761 q=4101) [ 8019.503111] rcu: rcu_sched kthread starved for 5995079 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 8019.509522] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8019.512231] rcu: RCU grace-period kthread stack dump: [ 8019.515662] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8019.521553] Call Trace: [ 8019.522696] __schedule+0x351/0xcb0 [ 8019.525018] schedule+0xc0/0x180 [ 8019.526764] schedule_timeout+0xb4/0x190 [ 8019.528667] ? do_raw_spin_unlock+0x75/0x190 [ 8019.530500] ? __next_timer_interrupt+0x160/0x160 [ 8019.534422] ? prepare_to_swait_event+0xf3/0x140 [ 8019.539661] rcu_gp_kthread+0x8dd/0x15b0 [ 8019.542142] ? force_qs_rnp+0x2c0/0x2c0 [ 8019.545264] kthread+0x1d1/0x200 [ 8019.546936] ? set_kthread_struct+0x70/0x70 [ 8019.552285] ret_from_fork+0x1f/0x30 [ 8019.555297] rcu: Stack dump where RCU GP kthread last ran: [ 8019.561536] NMI backtrace for cpu 0 [ 8019.562702] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8019.569189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8019.578929] Call Trace: [ 8019.584250] [ 8019.587464] ? dump_stack+0xbb/0x10e [ 8019.591452] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8019.596178] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8019.599397] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8019.605184] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8019.608467] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8019.612327] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 8019.616853] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 8019.623336] ? update_process_times+0x6d/0xc0 [ 8019.626744] ? tick_sched_handle.isra.18+0x27/0x90 [ 8019.631934] ? tick_sched_timer+0x53/0xb0 [ 8019.634524] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8019.636309] ? tick_sched_handle.isra.18+0x90/0x90 [ 8019.644346] ? hrtimer_interrupt+0x128/0x2c0 [ 8019.649830] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8019.654043] ? apic_timer_interrupt+0xf/0x20 [ 8019.656873] [ 8019.659151] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8019.673631] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8019.689778] ? dump_stack+0xf6/0x10e [ 8019.694797] ? do_raw_spin_lock+0x177/0x190 [ 8019.700076] ? _raw_spin_lock+0x21/0x30 [ 8019.701193] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8019.708365] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8019.715968] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8019.723393] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8019.726805] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8019.729925] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8019.736861] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8019.745411] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8019.750644] ? do_wait_intr+0xf0/0xf0 [ 8019.753461] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8019.758466] ? kthread+0x1d1/0x200 [ 8019.763775] ? set_kthread_struct+0x70/0x70 [ 8019.766703] ? ret_from_fork+0x1f/0x30 [ 8019.768787] NMI backtrace for cpu 0 [ 8019.772995] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8019.783444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8019.793506] Call Trace: [ 8019.795642] [ 8019.797345] ? dump_stack+0xbb/0x10e [ 8019.799400] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8019.801973] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8019.805654] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8019.807553] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8019.813464] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8019.817804] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8019.821369] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8019.824500] ? update_process_times+0x6d/0xc0 [ 8019.831776] ? tick_sched_handle.isra.18+0x27/0x90 [ 8019.840512] ? tick_sched_timer+0x53/0xb0 [ 8019.843015] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8019.846941] ? tick_sched_handle.isra.18+0x90/0x90 [ 8019.854835] ? hrtimer_interrupt+0x128/0x2c0 [ 8019.865158] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8019.866773] ? apic_timer_interrupt+0xf/0x20 [ 8019.878545] [ 8019.882862] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8019.887693] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8019.891266] ? dump_stack+0xf6/0x10e [ 8019.894388] ? do_raw_spin_lock+0x177/0x190 [ 8019.900380] ? _raw_spin_lock+0x21/0x30 [ 8019.906805] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8019.916013] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8019.921903] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8019.927478] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8019.930427] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8019.933702] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8019.935819] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8019.939386] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8019.941459] ? do_wait_intr+0xf0/0xf0 [ 8019.943143] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8019.947125] ? kthread+0x1d1/0x200 [ 8019.949952] ? set_kthread_struct+0x70/0x70 [ 8019.953181] ? ret_from_fork+0x1f/0x30 [ 8020.054436] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 8020.061949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8020.102163] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8020.121515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8020.131866] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8020.138608] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8020.150729] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8020.158693] RAX: 0000000000004685 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8020.162846] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8020.170966] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8020.175983] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8020.181959] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8020.186917] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8020.191935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8020.201386] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8020.207972] Call Trace: [ 8020.210416] [ 8020.213037] ? show_regs.cold.9+0x22/0x2f [ 8020.216710] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8020.219993] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8020.224029] ? watchdog+0x40/0x40 [ 8020.227022] ? hrtimer_interrupt+0x128/0x2c0 [ 8020.231034] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8020.236855] ? apic_timer_interrupt+0xf/0x20 [ 8020.241013] [ 8020.247518] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8020.255627] do_raw_spin_lock+0x177/0x190 [ 8020.263048] _raw_spin_lock+0x21/0x30 [ 8020.268381] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8020.272293] ? do_raw_spin_unlock+0x75/0x190 [ 8020.278063] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8020.281432] ? do_wait_intr+0xf0/0xf0 [ 8020.288230] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8020.292367] mdc_close+0x53b/0xd70 [mdc] [ 8020.294772] lmv_close+0x18d/0x490 [lmv] [ 8020.297770] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8020.305011] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8020.307714] ll_md_close+0x274/0x800 [lustre] [ 8020.311977] ll_file_release+0x128/0xc50 [lustre] [ 8020.316094] __fput+0x10d/0x440 [ 8020.317973] ____fput+0x12/0x20 [ 8020.320767] task_work_run+0xc6/0x110 [ 8020.322353] exit_to_usermode_loop+0x1dd/0x1f0 [ 8020.325860] do_syscall_64+0x3d6/0x440 [ 8020.336834] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8020.341139] RIP: 0033:0x7fdc0afbfcd8 [ 8020.343266] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8020.353056] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8020.360217] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8020.362019] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8020.379247] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8020.383520] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8020.393370] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8032.051188] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8032.055899] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8032.090353] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8032.098942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8032.104275] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8032.108780] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8032.112925] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8032.124728] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8032.129766] RAX: 0000000000006008 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8032.134371] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8032.138809] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8032.143288] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8032.148017] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8032.152560] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8032.157152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8032.160118] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8032.163752] Call Trace: [ 8032.164737] [ 8032.165548] ? show_regs.cold.9+0x22/0x2f [ 8032.168380] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8032.171735] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8032.175155] ? watchdog+0x40/0x40 [ 8032.177508] ? hrtimer_interrupt+0x128/0x2c0 [ 8032.179895] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8032.183089] ? apic_timer_interrupt+0xf/0x20 [ 8032.186349] [ 8032.187637] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8032.191095] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8032.194891] do_raw_spin_lock+0x177/0x190 [ 8032.197455] _raw_spin_lock+0x21/0x30 [ 8032.199815] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8032.204716] process_one_work+0x2c8/0x700 [ 8032.207299] worker_thread+0x296/0x6e0 [ 8032.211925] ? rescuer_thread+0x570/0x570 [ 8032.215196] kthread+0x1d1/0x200 [ 8032.218794] ? set_kthread_struct+0x70/0x70 [ 8032.226150] ret_from_fork+0x1f/0x30 [ 8034.783031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6009s! [ 8034.785980] Showing busy workqueues and worker pools: [ 8034.787709] workqueue events: flags=0x0 [ 8034.788991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8034.791814] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8034.794224] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8034.796940] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8034.801019] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8034.803996] pending: kfree_rcu_monitor [ 8034.805730] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8034.808446] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8034.812968] workqueue events_long: flags=0x0 [ 8034.814706] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8034.817138] pending: lu_objects_flush [obdclass] [ 8034.819112] workqueue events_unbound: flags=0x2 [ 8034.820840] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8034.823255] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8034.830029] workqueue events_power_efficient: flags=0x80 [ 8034.834183] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8034.838773] pending: neigh_periodic_work, check_lifetime [ 8034.844409] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8034.850717] pending: do_cache_clean, neigh_periodic_work [ 8034.854400] workqueue mm_percpu_wq: flags=0x8 [ 8034.855934] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8034.861844] pending: lru_add_drain_per_cpu [ 8034.866678] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8034.873717] pending: vmstat_update, lru_add_drain_per_cpu [ 8034.877140] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8034.885827] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8034.892464] workqueue writeback: flags=0x4a [ 8034.896753] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8034.902647] pending: wb_workfn, wb_workfn [ 8034.906656] workqueue xprtiod: flags=0xa [ 8034.909761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8034.913912] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8034.920666] workqueue ipv6_addrconf: flags=0x40008 [ 8034.924482] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8034.930708] pending: addrconf_verify_work [ 8034.933553] workqueue knbd0-recv: flags=0x1a [ 8034.938216] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8034.943779] in-flight: 655:recv_work [ 8034.944827] workqueue ptlrpc_pinger: flags=0x2 [ 8034.950203] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8034.953388] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8034.959565] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6010s workers=4 idle: 36396 38 11188 [ 8034.966932] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8025s workers=2 idle: 658 [ 8044.052006] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8044.055250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8044.081514] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8044.090099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8044.095287] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8044.097493] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8044.107857] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8044.115038] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8044.121350] RDX: 000000000000624a RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8044.131924] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8044.135722] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8044.142690] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8044.147890] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8044.153482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8044.156148] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8044.160356] Call Trace: [ 8044.163243] [ 8044.165316] ? show_regs.cold.9+0x22/0x2f [ 8044.168850] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8044.171079] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8044.178019] ? watchdog+0x40/0x40 [ 8044.183848] ? hrtimer_interrupt+0x128/0x2c0 [ 8044.186592] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8044.190818] ? apic_timer_interrupt+0xf/0x20 [ 8044.197174] [ 8044.199711] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8044.205821] do_raw_spin_lock+0x177/0x190 [ 8044.207383] _raw_spin_lock+0x21/0x30 [ 8044.209355] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8044.216451] ? do_raw_spin_unlock+0x75/0x190 [ 8044.221350] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8044.227969] ? do_wait_intr+0xf0/0xf0 [ 8044.229447] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8044.235341] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8044.240906] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8044.244329] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8044.252122] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8044.257181] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8044.262196] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8044.270287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8044.276294] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8044.280450] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8044.283166] lmv_intent_open+0x338/0x1150 [lmv] [ 8044.285741] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8044.288952] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8044.291973] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8044.294094] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8044.296460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8044.299255] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8044.301872] ? do_raw_spin_unlock+0x75/0x190 [ 8044.303523] ? _raw_spin_unlock+0x12/0x30 [ 8044.307141] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8044.309078] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8044.312177] ll_atomic_open+0x36a/0xd70 [lustre] [ 8044.313939] lookup_open+0x356/0xb40 [ 8044.317071] do_last+0x313/0xfc0 [ 8044.319367] ? nd_jump_root+0xe5/0x160 [ 8044.321806] ? path_init+0x437/0x520 [ 8044.323539] path_openat+0xf7/0x500 [ 8044.325187] ? do_raw_spin_unlock+0x75/0x190 [ 8044.329087] do_filp_open+0x99/0x140 [ 8044.330883] ? getname_flags+0x6e/0x330 [ 8044.333488] ? __check_object_size+0xff/0x256 [ 8044.338100] ? do_raw_spin_unlock+0x75/0x190 [ 8044.341259] ? _raw_spin_unlock+0x12/0x30 [ 8044.343209] do_sys_openat2+0x2b4/0x410 [ 8044.345063] do_sys_open+0x73/0xa0 [ 8044.346481] __x64_sys_openat+0x24/0x30 [ 8044.354981] do_syscall_64+0xc1/0x440 [ 8044.358993] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8044.363066] RIP: 0033:0x7fdc0afbf332 [ 8044.364422] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8044.373393] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8044.381101] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8044.392389] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8044.399118] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8044.404567] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8044.409830] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8048.057106] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 8048.061094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8048.097336] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8048.103164] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8048.106803] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8048.109332] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8048.117170] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8048.120113] RAX: 0000000000007ee1 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8048.122483] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8048.126310] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8048.129121] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8048.132279] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8048.134803] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8048.137366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8048.139463] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8048.146874] Call Trace: [ 8048.148110] [ 8048.150079] ? show_regs.cold.9+0x22/0x2f [ 8048.152207] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8048.156439] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8048.160483] ? watchdog+0x40/0x40 [ 8048.164436] ? hrtimer_interrupt+0x128/0x2c0 [ 8048.168281] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8048.174413] ? apic_timer_interrupt+0xf/0x20 [ 8048.177488] [ 8048.178440] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8048.183670] do_raw_spin_lock+0x177/0x190 [ 8048.189266] _raw_spin_lock+0x21/0x30 [ 8048.191096] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8048.196858] ? do_raw_spin_unlock+0x75/0x190 [ 8048.200256] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8048.203236] ? do_wait_intr+0xf0/0xf0 [ 8048.205122] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8048.207788] mdc_close+0x53b/0xd70 [mdc] [ 8048.209329] lmv_close+0x18d/0x490 [lmv] [ 8048.211193] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8048.215924] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8048.220162] ll_md_close+0x274/0x800 [lustre] [ 8048.224192] ll_file_release+0x128/0xc50 [lustre] [ 8048.226976] __fput+0x10d/0x440 [ 8048.229036] ____fput+0x12/0x20 [ 8048.230285] task_work_run+0xc6/0x110 [ 8048.232297] exit_to_usermode_loop+0x1dd/0x1f0 [ 8048.235140] do_syscall_64+0x3d6/0x440 [ 8048.238913] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8048.244892] RIP: 0033:0x7fdc0afbfcd8 [ 8048.251966] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8048.269033] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8048.273803] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8048.277413] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8048.280119] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8048.285959] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8048.292123] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8060.048719] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8060.052979] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8060.053213] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8060.079900] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8060.079914] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8060.079917] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8060.079929] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8060.079933] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8060.079940] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8060.079942] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8060.079945] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8060.079947] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8060.079950] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8060.079954] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8060.079957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8060.079959] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8060.079968] Call Trace: [ 8060.079973] [ 8060.079978] ? show_regs.cold.9+0x22/0x2f [ 8060.079987] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8060.079993] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8060.079997] ? watchdog+0x40/0x40 [ 8060.080002] ? hrtimer_interrupt+0x128/0x2c0 [ 8060.080006] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8060.080013] ? apic_timer_interrupt+0xf/0x20 [ 8060.080018] [ 8060.080021] ? native_safe_halt+0x2e/0x50 [ 8060.080025] kvm_wait+0xcd/0xe0 [ 8060.080033] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8060.080040] ? dump_stack+0xf6/0x10e [ 8060.080046] do_raw_spin_lock+0x177/0x190 [ 8060.080051] _raw_spin_lock+0x21/0x30 [ 8060.080059] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8060.080725] obd_mod_free+0x46/0x140 [ptlrpc] [ 8060.080968] mdc_commit_open+0x91/0xb0 [mdc] [ 8060.081011] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8060.081248] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8060.081702] after_reply+0xe2c/0x1780 [ptlrpc] [ 8060.081933] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8060.082135] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8060.082651] ? do_wait_intr+0xf0/0xf0 [ 8060.082661] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8060.082895] kthread+0x1d1/0x200 [ 8060.082902] ? set_kthread_struct+0x70/0x70 [ 8060.082906] ret_from_fork+0x1f/0x30 [ 8060.296870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8060.315978] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8060.320537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8060.323909] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8060.326328] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8060.328739] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8060.340190] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8060.343190] RAX: 000000000000734e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8060.348824] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8060.357833] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8060.365040] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8060.370716] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8060.379539] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8060.388807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8060.391554] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8060.395526] Call Trace: [ 8060.396788] [ 8060.397706] ? show_regs.cold.9+0x22/0x2f [ 8060.399130] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8060.407208] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8060.411493] ? watchdog+0x40/0x40 [ 8060.413284] ? hrtimer_interrupt+0x128/0x2c0 [ 8060.416062] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8060.418653] ? apic_timer_interrupt+0xf/0x20 [ 8060.423182] [ 8060.423933] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8060.429556] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8060.431314] do_raw_spin_lock+0x177/0x190 [ 8060.436478] _raw_spin_lock+0x21/0x30 [ 8060.441475] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8060.450520] process_one_work+0x2c8/0x700 [ 8060.453256] worker_thread+0x296/0x6e0 [ 8060.457685] ? rescuer_thread+0x570/0x570 [ 8060.462632] kthread+0x1d1/0x200 [ 8060.464551] ? set_kthread_struct+0x70/0x70 [ 8060.467588] ret_from_fork+0x1f/0x30 [ 8064.993982] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6040s! [ 8065.001494] Showing busy workqueues and worker pools: [ 8065.003359] workqueue events: flags=0x0 [ 8065.004453] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8065.007336] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8065.012758] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8065.015591] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8065.020862] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8065.024809] pending: kfree_rcu_monitor [ 8065.027306] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8065.031440] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8065.035642] workqueue events_long: flags=0x0 [ 8065.037412] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8065.043551] pending: lu_objects_flush [obdclass] [ 8065.056859] workqueue events_unbound: flags=0x2 [ 8065.060974] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8065.064866] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8065.073502] workqueue events_power_efficient: flags=0x80 [ 8065.077645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8065.081487] pending: neigh_periodic_work, check_lifetime [ 8065.083476] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8065.095135] pending: do_cache_clean, neigh_periodic_work [ 8065.100697] workqueue mm_percpu_wq: flags=0x8 [ 8065.102323] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8065.118495] pending: lru_add_drain_per_cpu [ 8065.123453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8065.130080] pending: vmstat_update, lru_add_drain_per_cpu [ 8065.135418] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8065.138301] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8065.140553] workqueue writeback: flags=0x4a [ 8065.142963] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8065.149570] pending: wb_workfn, wb_workfn [ 8065.151182] workqueue xprtiod: flags=0xa [ 8065.152601] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8065.158570] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8065.166437] workqueue ipv6_addrconf: flags=0x40008 [ 8065.169871] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8065.172795] pending: addrconf_verify_work [ 8065.180588] workqueue knbd0-recv: flags=0x1a [ 8065.183775] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8065.187816] in-flight: 655:recv_work [ 8065.193829] workqueue ptlrpc_pinger: flags=0x2 [ 8065.203004] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8065.209450] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8065.213414] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6040s workers=4 idle: 36396 38 11188 [ 8065.219645] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8055s workers=2 idle: 658 [ 8072.052034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8072.054310] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8072.070370] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8072.074384] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8072.080339] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8072.085244] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8072.096261] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8072.098719] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8072.101079] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8072.103405] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8072.105760] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8072.107878] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8072.110108] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8072.112817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8072.114642] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8072.116848] Call Trace: [ 8072.117711] [ 8072.118358] ? show_regs.cold.9+0x22/0x2f [ 8072.119524] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8072.121321] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8072.122726] ? watchdog+0x40/0x40 [ 8072.123720] ? hrtimer_interrupt+0x128/0x2c0 [ 8072.125251] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8072.126912] ? apic_timer_interrupt+0xf/0x20 [ 8072.128406] [ 8072.129272] ? native_safe_halt+0x2e/0x50 [ 8072.130607] kvm_wait+0xcd/0xe0 [ 8072.131698] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8072.133707] do_raw_spin_lock+0x177/0x190 [ 8072.135077] _raw_spin_lock+0x21/0x30 [ 8072.136336] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8072.138361] ? do_raw_spin_unlock+0x75/0x190 [ 8072.139843] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8072.141773] ? do_wait_intr+0xf0/0xf0 [ 8072.143080] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8072.145111] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8072.147110] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8072.149330] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8072.151453] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8072.153636] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8072.155434] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8072.157261] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8072.159219] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8072.161448] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8072.163496] lmv_intent_open+0x338/0x1150 [lmv] [ 8072.165117] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8072.167054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8072.169088] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8072.170691] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8072.172528] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8072.174498] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8072.176619] ? do_raw_spin_unlock+0x75/0x190 [ 8072.178115] ? _raw_spin_unlock+0x12/0x30 [ 8072.179512] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8072.183278] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8072.186838] ll_atomic_open+0x36a/0xd70 [lustre] [ 8072.189806] lookup_open+0x356/0xb40 [ 8072.191224] do_last+0x313/0xfc0 [ 8072.192195] ? nd_jump_root+0xe5/0x160 [ 8072.193588] ? path_init+0x437/0x520 [ 8072.194844] path_openat+0xf7/0x500 [ 8072.196138] ? do_raw_spin_unlock+0x75/0x190 [ 8072.197865] do_filp_open+0x99/0x140 [ 8072.199475] ? getname_flags+0x6e/0x330 [ 8072.200865] ? __check_object_size+0xff/0x256 [ 8072.203440] ? do_raw_spin_unlock+0x75/0x190 [ 8072.205004] ? _raw_spin_unlock+0x12/0x30 [ 8072.206161] do_sys_openat2+0x2b4/0x410 [ 8072.208196] do_sys_open+0x73/0xa0 [ 8072.209919] __x64_sys_openat+0x24/0x30 [ 8072.211882] do_syscall_64+0xc1/0x440 [ 8072.213315] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8072.215348] RIP: 0033:0x7fdc0afbf332 [ 8072.216675] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8072.224126] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8072.227227] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8072.230930] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8072.233528] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8072.236362] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8072.239410] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8076.055168] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8076.062751] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8076.094305] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8076.105643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8076.111431] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8076.115685] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8076.125864] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8076.128088] RAX: 00000000000074a0 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8076.129886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8076.132311] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8076.134618] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8076.136328] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8076.137685] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8076.139584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8076.140736] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8076.142297] Call Trace: [ 8076.142885] [ 8076.143411] ? show_regs.cold.9+0x22/0x2f [ 8076.144187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8076.145169] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8076.146121] ? watchdog+0x40/0x40 [ 8076.146870] ? hrtimer_interrupt+0x128/0x2c0 [ 8076.147892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8076.148945] ? apic_timer_interrupt+0xf/0x20 [ 8076.150515] [ 8076.151281] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8076.153258] do_raw_spin_lock+0x177/0x190 [ 8076.154740] _raw_spin_lock+0x21/0x30 [ 8076.156173] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8076.158294] ? do_raw_spin_unlock+0x75/0x190 [ 8076.159562] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8076.161012] ? do_wait_intr+0xf0/0xf0 [ 8076.162267] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8076.164438] mdc_close+0x53b/0xd70 [mdc] [ 8076.165939] lmv_close+0x18d/0x490 [lmv] [ 8076.167422] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8076.169326] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8076.170826] ll_md_close+0x274/0x800 [lustre] [ 8076.172725] ll_file_release+0x128/0xc50 [lustre] [ 8076.174736] __fput+0x10d/0x440 [ 8076.175667] ____fput+0x12/0x20 [ 8076.176928] task_work_run+0xc6/0x110 [ 8076.178669] exit_to_usermode_loop+0x1dd/0x1f0 [ 8076.179854] do_syscall_64+0x3d6/0x440 [ 8076.181043] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8076.182800] RIP: 0033:0x7fdc0afbfcd8 [ 8076.183875] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8076.189735] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8076.191925] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8076.193802] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8076.196180] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8076.198448] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8076.200395] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8088.051468] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8088.058823] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8088.090365] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8088.098612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8088.105629] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8088.109200] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8088.112243] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8088.123740] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8088.127437] RAX: 0000000000007a83 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8088.131341] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8088.133941] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8088.136552] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8088.138782] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8088.141920] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8088.144936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8088.147805] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8088.151392] Call Trace: [ 8088.151933] [ 8088.152606] ? show_regs.cold.9+0x22/0x2f [ 8088.153829] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8088.156872] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8088.158697] ? watchdog+0x40/0x40 [ 8088.162888] ? hrtimer_interrupt+0x128/0x2c0 [ 8088.164071] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8088.168323] ? apic_timer_interrupt+0xf/0x20 [ 8088.171651] [ 8088.173836] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8088.180776] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8088.184830] do_raw_spin_lock+0x177/0x190 [ 8088.185853] _raw_spin_lock+0x21/0x30 [ 8088.193436] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8088.198823] process_one_work+0x2c8/0x700 [ 8088.204629] worker_thread+0x296/0x6e0 [ 8088.209631] ? rescuer_thread+0x570/0x570 [ 8088.212748] kthread+0x1d1/0x200 [ 8088.216411] ? set_kthread_struct+0x70/0x70 [ 8088.219915] ret_from_fork+0x1f/0x30 [ 8092.052505] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 8092.062306] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8092.089105] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8092.093833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8092.096605] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8092.098532] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8092.104180] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8092.109515] RAX: 0000000000006a16 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8092.113520] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8092.117464] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8092.119208] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8092.127046] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8092.131569] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8092.137375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8092.139596] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8092.144023] Call Trace: [ 8092.145485] [ 8092.147681] ? show_regs.cold.9+0x22/0x2f [ 8092.152090] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8092.154446] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8092.155834] ? watchdog+0x40/0x40 [ 8092.158703] ? hrtimer_interrupt+0x128/0x2c0 [ 8092.160395] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8092.161543] ? apic_timer_interrupt+0xf/0x20 [ 8092.162903] [ 8092.163798] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8092.165143] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8092.167956] ? dump_stack+0xf6/0x10e [ 8092.168878] do_raw_spin_lock+0x177/0x190 [ 8092.169822] _raw_spin_lock+0x21/0x30 [ 8092.170756] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8092.177646] obd_mod_free+0x46/0x140 [ptlrpc] [ 8092.181252] mdc_commit_open+0x91/0xb0 [mdc] [ 8092.183013] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8092.187407] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8092.191684] after_reply+0xe2c/0x1780 [ptlrpc] [ 8092.194916] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8092.198164] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8092.200451] ? do_wait_intr+0xf0/0xf0 [ 8092.201608] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8092.205549] kthread+0x1d1/0x200 [ 8092.206593] ? set_kthread_struct+0x70/0x70 [ 8092.208937] ret_from_fork+0x1f/0x30 [ 8095.711106] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6070s! [ 8095.715113] Showing busy workqueues and worker pools: [ 8095.717080] workqueue events: flags=0x0 [ 8095.718427] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8095.722887] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8095.729539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8095.736250] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8095.742033] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8095.744451] pending: kfree_rcu_monitor [ 8095.746424] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8095.748655] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8095.756438] workqueue events_long: flags=0x0 [ 8095.761429] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8095.767673] pending: lu_objects_flush [obdclass] [ 8095.770515] workqueue events_unbound: flags=0x2 [ 8095.772526] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8095.774597] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8095.783024] workqueue events_power_efficient: flags=0x80 [ 8095.786324] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8095.792376] pending: neigh_periodic_work, check_lifetime [ 8095.795607] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8095.799623] pending: do_cache_clean, neigh_periodic_work [ 8095.802895] workqueue mm_percpu_wq: flags=0x8 [ 8095.806858] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8095.810173] pending: lru_add_drain_per_cpu [ 8095.813597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8095.817604] pending: vmstat_update, lru_add_drain_per_cpu [ 8095.820453] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8095.825218] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8095.834417] workqueue writeback: flags=0x4a [ 8095.839936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8095.842641] pending: wb_workfn, wb_workfn [ 8095.844475] workqueue xprtiod: flags=0xa [ 8095.845815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8095.848004] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8095.850508] workqueue ipv6_addrconf: flags=0x40008 [ 8095.854268] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8095.860834] pending: addrconf_verify_work [ 8095.863963] workqueue knbd0-recv: flags=0x1a [ 8095.866344] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8095.882102] in-flight: 655:recv_work [ 8095.886057] workqueue ptlrpc_pinger: flags=0x2 [ 8095.891351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8095.896237] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8095.903383] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6070s workers=4 idle: 36396 38 11188 [ 8095.908055] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8086s workers=2 idle: 658 [ 8100.054155] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8100.062010] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8100.101063] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8100.111060] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8100.118400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8100.121824] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8100.129204] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8100.133665] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8100.136581] RDX: 0000000000005d31 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8100.139995] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8100.143927] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8100.147171] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8100.150537] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8100.153711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8100.156121] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8100.158496] Call Trace: [ 8100.159331] [ 8100.160164] ? show_regs.cold.9+0x22/0x2f [ 8100.161847] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8100.163820] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8100.165454] ? watchdog+0x40/0x40 [ 8100.166447] ? hrtimer_interrupt+0x128/0x2c0 [ 8100.168041] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8100.169777] ? apic_timer_interrupt+0xf/0x20 [ 8100.171014] [ 8100.171727] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8100.173243] do_raw_spin_lock+0x177/0x190 [ 8100.174577] _raw_spin_lock+0x21/0x30 [ 8100.175740] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8100.177499] ? do_raw_spin_unlock+0x75/0x190 [ 8100.178808] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8100.180863] ? do_wait_intr+0xf0/0xf0 [ 8100.182057] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8100.186084] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8100.190243] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8100.192973] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8100.195385] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8100.199010] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8100.201533] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8100.206206] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8100.212738] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8100.217474] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8100.221117] lmv_intent_open+0x338/0x1150 [lmv] [ 8100.224076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8100.229847] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8100.235279] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8100.240233] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8100.244979] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8100.247921] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8100.250564] ? do_raw_spin_unlock+0x75/0x190 [ 8100.254870] ? _raw_spin_unlock+0x12/0x30 [ 8100.256899] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8100.261938] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8100.266302] ll_atomic_open+0x36a/0xd70 [lustre] [ 8100.271566] lookup_open+0x356/0xb40 [ 8100.273094] do_last+0x313/0xfc0 [ 8100.274531] ? nd_jump_root+0xe5/0x160 [ 8100.277222] ? path_init+0x437/0x520 [ 8100.282053] path_openat+0xf7/0x500 [ 8100.285764] ? do_raw_spin_unlock+0x75/0x190 [ 8100.289256] do_filp_open+0x99/0x140 [ 8100.294520] ? getname_flags+0x6e/0x330 [ 8100.300174] ? __check_object_size+0xff/0x256 [ 8100.304445] ? do_raw_spin_unlock+0x75/0x190 [ 8100.308282] ? _raw_spin_unlock+0x12/0x30 [ 8100.310161] do_sys_openat2+0x2b4/0x410 [ 8100.312410] do_sys_open+0x73/0xa0 [ 8100.314071] __x64_sys_openat+0x24/0x30 [ 8100.317791] do_syscall_64+0xc1/0x440 [ 8100.321539] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8100.324438] RIP: 0033:0x7fdc0afbf332 [ 8100.326450] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8100.344592] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8100.350429] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8100.359348] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8100.369615] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8100.379407] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8100.385506] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8104.054614] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8104.059946] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8104.116595] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8104.130540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8104.133932] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8104.142333] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8104.159771] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8104.169763] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8104.173112] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8104.179216] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8104.185033] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8104.190970] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8104.197756] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8104.204540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8104.206687] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8104.210874] Call Trace: [ 8104.211931] [ 8104.212862] ? show_regs.cold.9+0x22/0x2f [ 8104.214324] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8104.216306] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8104.221067] ? watchdog+0x40/0x40 [ 8104.222765] ? hrtimer_interrupt+0x128/0x2c0 [ 8104.226266] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8104.228878] ? apic_timer_interrupt+0xf/0x20 [ 8104.232357] [ 8104.236447] ? native_safe_halt+0x2e/0x50 [ 8104.237420] kvm_wait+0xcd/0xe0 [ 8104.238395] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8104.242672] do_raw_spin_lock+0x177/0x190 [ 8104.244063] _raw_spin_lock+0x21/0x30 [ 8104.247774] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8104.250228] ? do_raw_spin_unlock+0x75/0x190 [ 8104.254480] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8104.256497] ? do_wait_intr+0xf0/0xf0 [ 8104.257685] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8104.261996] mdc_close+0x53b/0xd70 [mdc] [ 8104.264210] lmv_close+0x18d/0x490 [lmv] [ 8104.265943] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8104.267705] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8104.269058] ll_md_close+0x274/0x800 [lustre] [ 8104.270545] ll_file_release+0x128/0xc50 [lustre] [ 8104.272039] __fput+0x10d/0x440 [ 8104.273220] ____fput+0x12/0x20 [ 8104.274266] task_work_run+0xc6/0x110 [ 8104.275432] exit_to_usermode_loop+0x1dd/0x1f0 [ 8104.277077] do_syscall_64+0x3d6/0x440 [ 8104.280047] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8104.281592] RIP: 0033:0x7fdc0afbfcd8 [ 8104.282696] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8104.289025] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8104.291426] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8104.293926] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8104.296505] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8104.299042] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8104.301710] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8116.051026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8116.054148] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8116.068869] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8116.072313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8116.074900] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8116.081853] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8116.084188] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8116.090894] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8116.093756] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8116.096630] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8116.098938] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8116.101073] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8116.103157] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8116.109078] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8116.111999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8116.114241] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8116.116922] Call Trace: [ 8116.118136] [ 8116.118852] ? show_regs.cold.9+0x22/0x2f [ 8116.120347] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8116.122164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8116.123979] ? watchdog+0x40/0x40 [ 8116.125340] ? hrtimer_interrupt+0x128/0x2c0 [ 8116.126796] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8116.128131] ? apic_timer_interrupt+0xf/0x20 [ 8116.129931] [ 8116.130653] ? native_safe_halt+0x2e/0x50 [ 8116.132654] kvm_wait+0xcd/0xe0 [ 8116.133806] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8116.135542] do_raw_spin_lock+0x177/0x190 [ 8116.136968] _raw_spin_lock+0x21/0x30 [ 8116.137929] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8116.139558] process_one_work+0x2c8/0x700 [ 8116.141278] worker_thread+0x296/0x6e0 [ 8116.143009] ? rescuer_thread+0x570/0x570 [ 8116.144519] kthread+0x1d1/0x200 [ 8116.145425] ? set_kthread_struct+0x70/0x70 [ 8116.147210] ret_from_fork+0x1f/0x30 [ 8120.048028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8120.052196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8120.097474] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8120.109754] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8120.120260] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8120.129976] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8120.155084] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8120.166683] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8120.174658] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8120.180593] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8120.185989] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8120.190813] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8120.196579] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8120.201003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8120.204263] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8120.207907] Call Trace: [ 8120.209682] [ 8120.211183] ? show_regs.cold.9+0x22/0x2f [ 8120.214916] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8120.220641] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8120.224579] ? watchdog+0x40/0x40 [ 8120.228028] ? hrtimer_interrupt+0x128/0x2c0 [ 8120.234637] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8120.241256] ? apic_timer_interrupt+0xf/0x20 [ 8120.244254] [ 8120.252033] ? native_safe_halt+0x2e/0x50 [ 8120.261891] kvm_wait+0xcd/0xe0 [ 8120.268885] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8120.273783] ? dump_stack+0xf6/0x10e [ 8120.277516] do_raw_spin_lock+0x177/0x190 [ 8120.292724] _raw_spin_lock+0x21/0x30 [ 8120.301039] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8120.316778] obd_mod_free+0x46/0x140 [ptlrpc] [ 8120.320415] mdc_commit_open+0x91/0xb0 [mdc] [ 8120.323928] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8120.326199] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8120.332081] after_reply+0xe2c/0x1780 [ptlrpc] [ 8120.340166] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8120.343641] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8120.348839] ? do_wait_intr+0xf0/0xf0 [ 8120.353074] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8120.364446] kthread+0x1d1/0x200 [ 8120.369623] ? set_kthread_struct+0x70/0x70 [ 8120.373192] ret_from_fork+0x1f/0x30 [ 8125.919030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6100s! [ 8125.924585] Showing busy workqueues and worker pools: [ 8125.926614] workqueue events: flags=0x0 [ 8125.928052] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8125.931044] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8125.934265] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8125.936778] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8125.946683] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8125.953542] pending: kfree_rcu_monitor [ 8125.958620] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8125.962248] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8125.970779] workqueue events_long: flags=0x0 [ 8125.972560] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8125.976355] pending: lu_objects_flush [obdclass] [ 8125.980092] workqueue events_unbound: flags=0x2 [ 8125.982151] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8125.984736] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8125.989718] workqueue events_power_efficient: flags=0x80 [ 8125.991952] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8125.996377] pending: neigh_periodic_work, check_lifetime [ 8125.999658] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8126.004173] pending: do_cache_clean, neigh_periodic_work [ 8126.007841] workqueue mm_percpu_wq: flags=0x8 [ 8126.010142] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8126.014822] pending: lru_add_drain_per_cpu [ 8126.016771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8126.021389] pending: vmstat_update, lru_add_drain_per_cpu [ 8126.025147] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8126.029446] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8126.033640] workqueue writeback: flags=0x4a [ 8126.035946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8126.038639] pending: wb_workfn, wb_workfn [ 8126.040551] workqueue xprtiod: flags=0xa [ 8126.042569] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8126.045345] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8126.050455] workqueue ipv6_addrconf: flags=0x40008 [ 8126.053064] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8126.059318] pending: addrconf_verify_work [ 8126.062904] workqueue knbd0-recv: flags=0x1a [ 8126.067465] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8126.071929] in-flight: 655:recv_work [ 8126.073793] workqueue ptlrpc_pinger: flags=0x2 [ 8126.077408] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8126.082110] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8126.086882] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6101s workers=4 idle: 36396 38 11188 [ 8126.093517] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8116s workers=2 idle: 658 [ 8128.052095] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8128.055949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8128.105452] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8128.117954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8128.127969] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8128.134295] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8128.154467] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8128.162579] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8128.173909] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8128.182910] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8128.190901] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8128.198666] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8128.206647] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8128.214964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8128.222201] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8128.226516] Call Trace: [ 8128.229525] [ 8128.230602] ? show_regs.cold.9+0x22/0x2f [ 8128.234908] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8128.239411] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8128.245988] ? watchdog+0x40/0x40 [ 8128.247652] ? hrtimer_interrupt+0x128/0x2c0 [ 8128.254275] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8128.257270] ? apic_timer_interrupt+0xf/0x20 [ 8128.259700] [ 8128.262645] ? native_safe_halt+0x2e/0x50 [ 8128.266634] kvm_wait+0xcd/0xe0 [ 8128.270391] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8128.277024] do_raw_spin_lock+0x177/0x190 [ 8128.281514] _raw_spin_lock+0x21/0x30 [ 8128.284674] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8128.292147] ? do_raw_spin_unlock+0x75/0x190 [ 8128.296811] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8128.302667] ? do_wait_intr+0xf0/0xf0 [ 8128.304751] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8128.307959] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8128.315312] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8128.322376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8128.328520] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8128.335653] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8128.339759] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8128.347661] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8128.354630] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8128.364057] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8128.370380] lmv_intent_open+0x338/0x1150 [lmv] [ 8128.373943] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8128.377476] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8128.384654] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8128.390889] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8128.395467] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8128.403745] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8128.411237] ? do_raw_spin_unlock+0x75/0x190 [ 8128.412839] ? _raw_spin_unlock+0x12/0x30 [ 8128.417140] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8128.424670] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8128.427510] ll_atomic_open+0x36a/0xd70 [lustre] [ 8128.429609] lookup_open+0x356/0xb40 [ 8128.433344] do_last+0x313/0xfc0 [ 8128.435819] ? nd_jump_root+0xe5/0x160 [ 8128.440813] ? path_init+0x437/0x520 [ 8128.445334] path_openat+0xf7/0x500 [ 8128.448064] ? do_raw_spin_unlock+0x75/0x190 [ 8128.452087] do_filp_open+0x99/0x140 [ 8128.456051] ? getname_flags+0x6e/0x330 [ 8128.460022] ? __check_object_size+0xff/0x256 [ 8128.464342] ? do_raw_spin_unlock+0x75/0x190 [ 8128.470633] ? _raw_spin_unlock+0x12/0x30 [ 8128.477190] do_sys_openat2+0x2b4/0x410 [ 8128.482546] do_sys_open+0x73/0xa0 [ 8128.486707] __x64_sys_openat+0x24/0x30 [ 8128.493136] do_syscall_64+0xc1/0x440 [ 8128.497000] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8128.503129] RIP: 0033:0x7fdc0afbf332 [ 8128.507850] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8128.527453] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8128.535081] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8128.544443] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8128.550765] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8128.561383] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8128.572209] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8132.054068] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8132.057496] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8132.079662] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8132.085927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8132.089843] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8132.092467] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8132.101642] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8132.103882] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8132.106838] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8132.109769] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8132.113280] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8132.116268] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8132.118761] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8132.121906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8132.124651] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8132.126881] Call Trace: [ 8132.128045] [ 8132.128963] ? show_regs.cold.9+0x22/0x2f [ 8132.131047] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8132.133797] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8132.136360] ? watchdog+0x40/0x40 [ 8132.138150] ? hrtimer_interrupt+0x128/0x2c0 [ 8132.140425] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8132.144324] ? apic_timer_interrupt+0xf/0x20 [ 8132.147764] [ 8132.149011] ? native_safe_halt+0x2e/0x50 [ 8132.150695] kvm_wait+0xcd/0xe0 [ 8132.152355] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8132.155182] do_raw_spin_lock+0x177/0x190 [ 8132.157702] _raw_spin_lock+0x21/0x30 [ 8132.159750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8132.163682] ? do_raw_spin_unlock+0x75/0x190 [ 8132.166259] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8132.168912] ? do_wait_intr+0xf0/0xf0 [ 8132.170745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8132.173482] mdc_close+0x53b/0xd70 [mdc] [ 8132.175447] lmv_close+0x18d/0x490 [lmv] [ 8132.177573] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8132.180326] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8132.182865] ll_md_close+0x274/0x800 [lustre] [ 8132.185197] ll_file_release+0x128/0xc50 [lustre] [ 8132.187991] __fput+0x10d/0x440 [ 8132.189714] ____fput+0x12/0x20 [ 8132.192044] task_work_run+0xc6/0x110 [ 8132.193973] exit_to_usermode_loop+0x1dd/0x1f0 [ 8132.197299] do_syscall_64+0x3d6/0x440 [ 8132.199218] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8132.201990] RIP: 0033:0x7fdc0afbfcd8 [ 8132.203915] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8132.213272] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8132.216978] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8132.220633] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8132.224340] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8132.227744] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8132.231504] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8144.051066] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8144.058125] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8144.086444] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8144.091853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8144.098957] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8144.103048] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8144.110767] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8144.124629] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8144.129082] RAX: 0000000000003549 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8144.133380] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8144.139062] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8144.142266] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8144.147986] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8144.153974] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8144.159666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8144.161695] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8144.169831] Call Trace: [ 8144.173015] [ 8144.176123] ? show_regs.cold.9+0x22/0x2f [ 8144.179481] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8144.184341] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8144.186530] ? watchdog+0x40/0x40 [ 8144.187996] ? hrtimer_interrupt+0x128/0x2c0 [ 8144.190202] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8144.192201] ? apic_timer_interrupt+0xf/0x20 [ 8144.200769] [ 8144.201749] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8144.205327] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8144.214913] do_raw_spin_lock+0x177/0x190 [ 8144.215998] _raw_spin_lock+0x21/0x30 [ 8144.219264] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8144.222245] process_one_work+0x2c8/0x700 [ 8144.226099] worker_thread+0x296/0x6e0 [ 8144.231849] ? rescuer_thread+0x570/0x570 [ 8144.235076] kthread+0x1d1/0x200 [ 8144.236715] ? set_kthread_struct+0x70/0x70 [ 8144.238829] ret_from_fork+0x1f/0x30 [ 8152.050022] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 8152.054488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8152.083118] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8152.090036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8152.094458] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8152.097455] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8152.106428] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8152.110146] RAX: 0000000000003b0b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8152.114071] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8152.122154] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8152.130074] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8152.133251] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8152.141422] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8152.146299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8152.151362] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8152.155048] Call Trace: [ 8152.156638] [ 8152.157429] ? show_regs.cold.9+0x22/0x2f [ 8152.162132] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8152.166095] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8152.168174] ? watchdog+0x40/0x40 [ 8152.169591] ? hrtimer_interrupt+0x128/0x2c0 [ 8152.171791] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8152.173549] ? apic_timer_interrupt+0xf/0x20 [ 8152.176021] [ 8152.178972] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8152.181458] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8152.183642] ? dump_stack+0xf6/0x10e [ 8152.186252] do_raw_spin_lock+0x177/0x190 [ 8152.188310] _raw_spin_lock+0x21/0x30 [ 8152.193489] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8152.197410] obd_mod_free+0x46/0x140 [ptlrpc] [ 8152.200578] mdc_commit_open+0x91/0xb0 [mdc] [ 8152.202955] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8152.205406] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8152.211628] after_reply+0xe2c/0x1780 [ptlrpc] [ 8152.215371] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8152.224017] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8152.229345] ? do_wait_intr+0xf0/0xf0 [ 8152.234354] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8152.236254] kthread+0x1d1/0x200 [ 8152.243554] ? set_kthread_struct+0x70/0x70 [ 8152.250899] ret_from_fork+0x1f/0x30 [ 8156.052006] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8156.056370] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8156.087289] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8156.094581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8156.099968] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8156.104433] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8156.117173] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8156.121044] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8156.126568] RDX: 0000000000005df8 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8156.127025] BUG: workqueue lockup - pool [ 8156.135245] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8156.135257] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8156.135260] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8156.135264] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8156.135267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8156.135270] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8156.135279] Call Trace: [ 8156.135287] [ 8156.135292] ? show_regs.cold.9+0x22/0x2f [ 8156.135306] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8156.135314] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8156.135319] ? watchdog+0x40/0x40 [ 8156.135323] ? hrtimer_interrupt+0x128/0x2c0 [ 8156.135327] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8156.135335] ? apic_timer_interrupt+0xf/0x20 [ 8156.135340] [ 8156.135344] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8156.135352] do_raw_spin_lock+0x177/0x190 [ 8156.135357] _raw_spin_lock+0x21/0x30 [ 8156.135364] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8156.137395] cpus=0-3 flags=0x4 nice=0 stuck for 6131s! [ 8156.137412] Showing busy workqueues and worker pools: [ 8156.137414] workqueue events: flags=0x0 [ 8156.137417] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8156.137426] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8156.137713] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8156.137722] pending: cpu_latency_work [ptlrpc] [ 8156.141349] ? do_raw_spin_unlock+0x75/0x190 [ 8156.141369] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8156.145403] , osc_grant_work_handler [osc] [ 8156.149511] ? do_wait_intr+0xf0/0xf0 [ 8156.155199] , key_garbage_collector [ 8156.155249] pwq 2: [ 8156.163484] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8156.167806] cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8156.167827] pending: kfree_rcu_monitor [ 8156.167877] pwq 0: [ 8156.170294] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8156.171553] cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8156.171566] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8156.173062] workqueue events_long: flags=0x0 [ 8156.173075] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8156.173085] pending: lu_objects_flush [obdclass] [ 8156.173243] workqueue events_unbound: flags=0x2 [ 8156.173245] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8156.173252] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8156.173272] workqueue events_power_efficient: flags=0x80 [ 8156.173274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8156.173279] pending: neigh_periodic_work, check_lifetime [ 8156.173291] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8156.173297] pending: do_cache_clean, neigh_periodic_work [ 8156.173308] workqueue mm_percpu_wq: flags=0x8 [ 8156.173373] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8156.173379] pending: lru_add_drain_per_cpu [ 8156.173391] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8156.173398] pending: vmstat_update, lru_add_drain_per_cpu [ 8156.173408] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8156.173415] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8156.173427] workqueue writeback: flags=0x4a [ 8156.173429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8156.173434] pending: wb_workfn, wb_workfn [ 8156.173447] workqueue xprtiod: flags=0xa [ 8156.173450] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8156.173455] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8156.173475] workqueue ipv6_addrconf: flags=0x40008 [ 8156.173478] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8156.173485] pending: addrconf_verify_work [ 8156.173497] workqueue knbd0-recv: flags=0x1a [ 8156.173499] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8156.173506] in-flight: 655:recv_work [ 8156.173516] workqueue ptlrpc_pinger: flags=0x2 [ 8156.173519] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8156.173524] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8156.173873] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6131s workers=4 idle: 36396 38 11188 [ 8156.173886] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8146s workers=2 idle: 658 [ 8156.496192] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8156.498911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8156.503032] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8156.506181] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8156.509083] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8156.512074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8156.515618] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8156.520403] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8156.526744] lmv_intent_open+0x338/0x1150 [lmv] [ 8156.528808] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8156.531527] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8156.533307] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8156.534550] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8156.536316] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8156.540334] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8156.542573] ? do_raw_spin_unlock+0x75/0x190 [ 8156.546323] ? _raw_spin_unlock+0x12/0x30 [ 8156.547932] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8156.550105] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8156.552577] ll_atomic_open+0x36a/0xd70 [lustre] [ 8156.554754] lookup_open+0x356/0xb40 [ 8156.555805] do_last+0x313/0xfc0 [ 8156.556644] ? nd_jump_root+0xe5/0x160 [ 8156.557832] ? path_init+0x437/0x520 [ 8156.559316] path_openat+0xf7/0x500 [ 8156.560882] ? do_raw_spin_unlock+0x75/0x190 [ 8156.562682] do_filp_open+0x99/0x140 [ 8156.563638] ? getname_flags+0x6e/0x330 [ 8156.564531] ? __check_object_size+0xff/0x256 [ 8156.571473] ? do_raw_spin_unlock+0x75/0x190 [ 8156.573245] ? _raw_spin_unlock+0x12/0x30 [ 8156.574882] do_sys_openat2+0x2b4/0x410 [ 8156.576384] do_sys_open+0x73/0xa0 [ 8156.579896] __x64_sys_openat+0x24/0x30 [ 8156.581457] do_syscall_64+0xc1/0x440 [ 8156.582970] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8156.585110] RIP: 0033:0x7fdc0afbf332 [ 8156.586615] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8156.593885] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8156.597046] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8156.599777] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8156.602578] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8156.605488] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8156.608357] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8160.054244] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8160.057890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8160.110066] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8160.117321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8160.122233] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8160.126261] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8160.136166] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8160.138639] RAX: 0000000000002f7e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8160.142425] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8160.146748] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8160.152515] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8160.163708] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8160.174500] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8160.187011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8160.189128] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8160.196063] Call Trace: [ 8160.198800] [ 8160.203597] ? show_regs.cold.9+0x22/0x2f [ 8160.206671] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8160.212354] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8160.218154] ? watchdog+0x40/0x40 [ 8160.220097] ? hrtimer_interrupt+0x128/0x2c0 [ 8160.226801] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8160.231135] ? apic_timer_interrupt+0xf/0x20 [ 8160.232916] [ 8160.240524] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8160.242972] do_raw_spin_lock+0x177/0x190 [ 8160.248485] _raw_spin_lock+0x21/0x30 [ 8160.250135] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8160.253648] ? do_raw_spin_unlock+0x75/0x190 [ 8160.259342] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8160.263067] ? do_wait_intr+0xf0/0xf0 [ 8160.266815] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8160.277016] mdc_close+0x53b/0xd70 [mdc] [ 8160.280577] lmv_close+0x18d/0x490 [lmv] [ 8160.283263] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8160.285258] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8160.292986] ll_md_close+0x274/0x800 [lustre] [ 8160.300959] ll_file_release+0x128/0xc50 [lustre] [ 8160.308510] __fput+0x10d/0x440 [ 8160.314720] ____fput+0x12/0x20 [ 8160.318316] task_work_run+0xc6/0x110 [ 8160.321027] exit_to_usermode_loop+0x1dd/0x1f0 [ 8160.323437] do_syscall_64+0x3d6/0x440 [ 8160.325877] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8160.327853] RIP: 0033:0x7fdc0afbfcd8 [ 8160.333751] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8160.345819] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8160.354964] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8160.360075] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8160.363028] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8160.366456] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8160.369184] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8172.051021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8172.054722] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8172.093516] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8172.099611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8172.108320] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8172.121845] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8172.132334] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8172.143759] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8172.147565] RAX: 000000000000254b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8172.153344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8172.158740] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8172.163174] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8172.167902] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8172.175092] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8172.179779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8172.182646] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8172.187592] Call Trace: [ 8172.189843] [ 8172.190739] ? show_regs.cold.9+0x22/0x2f [ 8172.192458] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8172.194705] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8172.196527] ? watchdog+0x40/0x40 [ 8172.197922] ? hrtimer_interrupt+0x128/0x2c0 [ 8172.199859] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8172.201899] ? apic_timer_interrupt+0xf/0x20 [ 8172.203746] [ 8172.204627] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8172.206920] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8172.209221] do_raw_spin_lock+0x177/0x190 [ 8172.211022] _raw_spin_lock+0x21/0x30 [ 8172.212519] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8172.214975] process_one_work+0x2c8/0x700 [ 8172.216631] worker_thread+0x296/0x6e0 [ 8172.218159] ? rescuer_thread+0x570/0x570 [ 8172.219688] kthread+0x1d1/0x200 [ 8172.221931] ? set_kthread_struct+0x70/0x70 [ 8172.223675] ret_from_fork+0x1f/0x30 [ 8184.048459] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8184.052468] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8184.057629] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8184.057713] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8184.057721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8184.057724] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8184.057739] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8184.057744] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8184.057748] RAX: 000000000000251e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8184.057751] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8184.057753] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8184.057756] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8184.057758] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8184.057762] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8184.057765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8184.057767] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8184.057777] Call Trace: [ 8184.057782] [ 8184.057788] ? show_regs.cold.9+0x22/0x2f [ 8184.057799] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8184.057807] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8184.057812] ? watchdog+0x40/0x40 [ 8184.057817] ? hrtimer_interrupt+0x128/0x2c0 [ 8184.057822] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8184.057831] ? apic_timer_interrupt+0xf/0x20 [ 8184.057836] [ 8184.057839] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8184.057844] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8184.057849] ? dump_stack+0xf6/0x10e [ 8184.057855] do_raw_spin_lock+0x177/0x190 [ 8184.057860] _raw_spin_lock+0x21/0x30 [ 8184.057866] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8184.058173] obd_mod_free+0x46/0x140 [ptlrpc] [ 8184.063733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8184.063819] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8184.063827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8184.063830] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8184.063850] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8184.063855] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8184.063865] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8184.063867] RDX: 0000000000007e89 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8184.063870] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8184.063872] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8184.063875] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8184.063878] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8184.063882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8184.063890] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8184.063899] Call Trace: [ 8184.063905] [ 8184.063914] ? show_regs.cold.9+0x22/0x2f [ 8184.063924] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8184.063930] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8184.063935] ? watchdog+0x40/0x40 [ 8184.063938] ? hrtimer_interrupt+0x128/0x2c0 [ 8184.063942] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8184.063950] ? apic_timer_interrupt+0xf/0x20 [ 8184.063954] [ 8184.063957] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8184.063963] do_raw_spin_lock+0x177/0x190 [ 8184.063967] _raw_spin_lock+0x21/0x30 [ 8184.063973] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8184.116526] mdc_commit_open+0x91/0xb0 [mdc] [ 8184.122485] ? do_raw_spin_unlock+0x75/0x190 [ 8184.122506] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8184.128895] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8184.131235] ? do_wait_intr+0xf0/0xf0 [ 8184.131258] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8184.132155] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8184.144027] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8184.147919] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8184.156571] after_reply+0xe2c/0x1780 [ptlrpc] [ 8184.156874] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8184.162054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8184.162983] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8184.163022] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8184.163054] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8184.163089] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8184.163210] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8184.169021] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8184.172369] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8184.172646] lmv_intent_open+0x338/0x1150 [lmv] [ 8184.172682] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8184.172965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8184.173049] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8184.173074] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8184.173162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8184.179962] ? do_wait_intr+0xf0/0xf0 [ 8184.184318] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8184.184464] ? do_raw_spin_unlock+0x75/0x190 [ 8184.184474] ? _raw_spin_unlock+0x12/0x30 [ 8184.184484] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8184.184518] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8184.184545] ll_atomic_open+0x36a/0xd70 [lustre] [ 8184.184669] lookup_open+0x356/0xb40 [ 8184.184678] do_last+0x313/0xfc0 [ 8184.184683] ? nd_jump_root+0xe5/0x160 [ 8184.184688] ? path_init+0x437/0x520 [ 8184.184692] path_openat+0xf7/0x500 [ 8184.184697] ? do_raw_spin_unlock+0x75/0x190 [ 8184.184702] do_filp_open+0x99/0x140 [ 8184.184708] ? getname_flags+0x6e/0x330 [ 8184.184712] ? __check_object_size+0xff/0x256 [ 8184.184719] ? do_raw_spin_unlock+0x75/0x190 [ 8184.184723] ? _raw_spin_unlock+0x12/0x30 [ 8184.184732] do_sys_openat2+0x2b4/0x410 [ 8184.184736] do_sys_open+0x73/0xa0 [ 8184.184740] __x64_sys_openat+0x24/0x30 [ 8184.184744] do_syscall_64+0xc1/0x440 [ 8184.184751] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8184.184757] RIP: 0033:0x7fdc0afbf332 [ 8184.184765] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8184.184769] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8184.184774] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8184.184777] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8184.184779] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8184.184782] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8184.184784] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8184.637882] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8184.640524] kthread+0x1d1/0x200 [ 8184.642339] ? set_kthread_struct+0x70/0x70 [ 8184.644064] ret_from_fork+0x1f/0x30 [ 8186.335516] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6161s! [ 8186.343251] Showing busy workqueues and worker pools: [ 8186.349033] workqueue events: flags=0x0 [ 8186.357756] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8186.361918] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8186.365750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8186.369871] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8186.375806] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8186.379971] pending: kfree_rcu_monitor [ 8186.385672] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8186.390027] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8186.397725] workqueue events_long: flags=0x0 [ 8186.399853] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8186.406329] pending: lu_objects_flush [obdclass] [ 8186.408219] workqueue events_unbound: flags=0x2 [ 8186.413854] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8186.419735] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8186.425599] workqueue events_power_efficient: flags=0x80 [ 8186.428821] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8186.434429] pending: neigh_periodic_work, check_lifetime [ 8186.438353] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8186.443492] pending: do_cache_clean, neigh_periodic_work [ 8186.457870] workqueue mm_percpu_wq: flags=0x8 [ 8186.463491] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8186.469943] pending: lru_add_drain_per_cpu [ 8186.472438] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8186.476451] pending: vmstat_update, lru_add_drain_per_cpu [ 8186.479376] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8186.483455] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8186.487202] workqueue writeback: flags=0x4a [ 8186.488759] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8186.497062] pending: wb_workfn, wb_workfn [ 8186.502693] workqueue xprtiod: flags=0xa [ 8186.507040] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8186.512018] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8186.521960] workqueue ipv6_addrconf: flags=0x40008 [ 8186.527703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8186.532548] pending: addrconf_verify_work [ 8186.534935] workqueue knbd0-recv: flags=0x1a [ 8186.537889] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8186.541496] in-flight: 655:recv_work [ 8186.543527] workqueue ptlrpc_pinger: flags=0x2 [ 8186.545702] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8186.548758] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8186.551398] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6161s workers=4 idle: 36396 38 11188 [ 8186.556570] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8177s workers=2 idle: 658 [ 8188.054009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8188.059312] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8188.101979] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8188.113930] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8188.117626] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8188.120013] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8188.137262] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8188.140922] RAX: 0000000000005e2f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8188.143918] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8188.148827] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8188.151254] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8188.153801] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8188.158349] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8188.164054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8188.167567] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8188.170297] Call Trace: [ 8188.171500] [ 8188.172411] ? show_regs.cold.9+0x22/0x2f [ 8188.175050] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8188.180398] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8188.184531] ? watchdog+0x40/0x40 [ 8188.186332] ? hrtimer_interrupt+0x128/0x2c0 [ 8188.190360] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8188.191863] ? apic_timer_interrupt+0xf/0x20 [ 8188.194937] [ 8188.196654] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8188.200375] do_raw_spin_lock+0x177/0x190 [ 8188.202244] _raw_spin_lock+0x21/0x30 [ 8188.205398] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8188.208538] ? do_raw_spin_unlock+0x75/0x190 [ 8188.212258] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8188.219057] ? do_wait_intr+0xf0/0xf0 [ 8188.221952] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8188.224975] mdc_close+0x53b/0xd70 [mdc] [ 8188.227887] lmv_close+0x18d/0x490 [lmv] [ 8188.230914] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8188.232533] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8188.234799] ll_md_close+0x274/0x800 [lustre] [ 8188.236906] ll_file_release+0x128/0xc50 [lustre] [ 8188.239030] __fput+0x10d/0x440 [ 8188.247476] ____fput+0x12/0x20 [ 8188.248343] task_work_run+0xc6/0x110 [ 8188.252747] exit_to_usermode_loop+0x1dd/0x1f0 [ 8188.255559] do_syscall_64+0x3d6/0x440 [ 8188.259062] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8188.262061] RIP: 0033:0x7fdc0afbfcd8 [ 8188.265723] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8188.274698] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8188.280710] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8188.282867] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8188.299870] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8188.307833] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8188.313730] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8199.495893] rcu: INFO: rcu_sched self-detected stall on CPU [ 8199.498669] rcu: 0-...!: (3876596 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 8199.507681] (t=6180152 jiffies g=270761 q=4109) [ 8199.510391] rcu: rcu_sched kthread starved for 6175084 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 8199.519832] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8199.531539] rcu: RCU grace-period kthread stack dump: [ 8199.533326] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8199.542944] Call Trace: [ 8199.545437] __schedule+0x351/0xcb0 [ 8199.548296] schedule+0xc0/0x180 [ 8199.550007] schedule_timeout+0xb4/0x190 [ 8199.552789] ? do_raw_spin_unlock+0x75/0x190 [ 8199.561133] ? __next_timer_interrupt+0x160/0x160 [ 8199.564739] ? prepare_to_swait_event+0xf3/0x140 [ 8199.568061] rcu_gp_kthread+0x8dd/0x15b0 [ 8199.570364] ? force_qs_rnp+0x2c0/0x2c0 [ 8199.574285] kthread+0x1d1/0x200 [ 8199.576075] ? set_kthread_struct+0x70/0x70 [ 8199.581606] ret_from_fork+0x1f/0x30 [ 8199.582781] rcu: Stack dump where RCU GP kthread last ran: [ 8199.585886] NMI backtrace for cpu 0 [ 8199.591350] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8199.597138] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8199.603808] Call Trace: [ 8199.605272] [ 8199.606354] ? dump_stack+0xbb/0x10e [ 8199.610885] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8199.615577] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8199.618725] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8199.621427] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8199.627942] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8199.632430] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 8199.634784] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 8199.641602] ? update_process_times+0x6d/0xc0 [ 8199.644654] ? tick_sched_handle.isra.18+0x27/0x90 [ 8199.647382] ? tick_sched_timer+0x53/0xb0 [ 8199.655189] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8199.656570] ? tick_sched_handle.isra.18+0x90/0x90 [ 8199.661523] ? hrtimer_interrupt+0x128/0x2c0 [ 8199.662743] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8199.669789] ? apic_timer_interrupt+0xf/0x20 [ 8199.675466] [ 8199.681909] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8199.683438] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8199.688802] ? dump_stack+0xf6/0x10e [ 8199.690204] ? do_raw_spin_lock+0x177/0x190 [ 8199.695680] ? _raw_spin_lock+0x21/0x30 [ 8199.697514] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8199.705085] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8199.714709] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8199.716323] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8199.724435] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8199.732669] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8199.737127] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8199.741768] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8199.744759] ? do_wait_intr+0xf0/0xf0 [ 8199.747039] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8199.751340] ? kthread+0x1d1/0x200 [ 8199.753173] ? set_kthread_struct+0x70/0x70 [ 8199.757877] ? ret_from_fork+0x1f/0x30 [ 8199.759580] NMI backtrace for cpu 0 [ 8199.766085] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8199.774798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8199.785533] Call Trace: [ 8199.789197] [ 8199.789978] ? dump_stack+0xbb/0x10e [ 8199.792184] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8199.795173] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8199.798014] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8199.802990] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8199.808791] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8199.811260] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8199.816333] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8199.822028] ? update_process_times+0x6d/0xc0 [ 8199.824331] ? tick_sched_handle.isra.18+0x27/0x90 [ 8199.827813] ? tick_sched_timer+0x53/0xb0 [ 8199.829192] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8199.832187] ? tick_sched_handle.isra.18+0x90/0x90 [ 8199.834839] ? hrtimer_interrupt+0x128/0x2c0 [ 8199.837015] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8199.846352] ? apic_timer_interrupt+0xf/0x20 [ 8199.851195] [ 8199.853548] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8199.858276] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8199.862696] ? dump_stack+0xf6/0x10e [ 8199.863667] ? do_raw_spin_lock+0x177/0x190 [ 8199.865787] ? _raw_spin_lock+0x21/0x30 [ 8199.868940] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8199.878073] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8199.879885] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8199.881918] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8199.884348] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8199.889276] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8199.890933] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8199.896622] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8199.900865] ? do_wait_intr+0xf0/0xf0 [ 8199.903425] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8199.908726] ? kthread+0x1d1/0x200 [ 8199.911384] ? set_kthread_struct+0x70/0x70 [ 8199.913730] ? ret_from_fork+0x1f/0x30 [ 8200.052477] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8200.062345] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8200.132792] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8200.138835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8200.147595] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8200.154309] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8200.157631] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8200.191754] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8200.204538] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8200.207987] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8200.213581] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8200.231625] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8200.245758] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8200.255402] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8200.258472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8200.263225] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8200.272540] Call Trace: [ 8200.274854] [ 8200.276952] ? show_regs.cold.9+0x22/0x2f [ 8200.280265] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8200.283780] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8200.287713] ? watchdog+0x40/0x40 [ 8200.290834] ? hrtimer_interrupt+0x128/0x2c0 [ 8200.293626] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8200.297246] ? apic_timer_interrupt+0xf/0x20 [ 8200.299128] [ 8200.300065] ? native_safe_halt+0x2e/0x50 [ 8200.305841] kvm_wait+0xcd/0xe0 [ 8200.309827] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8200.313836] do_raw_spin_lock+0x177/0x190 [ 8200.319497] _raw_spin_lock+0x21/0x30 [ 8200.324527] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8200.328417] process_one_work+0x2c8/0x700 [ 8200.330559] worker_thread+0x296/0x6e0 [ 8200.332346] ? rescuer_thread+0x570/0x570 [ 8200.334443] kthread+0x1d1/0x200 [ 8200.335817] ? set_kthread_struct+0x70/0x70 [ 8200.337685] ret_from_fork+0x1f/0x30 [ 8212.053363] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 8212.056302] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8212.081469] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8212.086096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8212.089345] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8212.091079] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8212.103176] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8212.106947] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8212.112408] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8212.117303] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8212.122670] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8212.127593] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8212.130066] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8212.138629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8212.144370] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8212.158706] Call Trace: [ 8212.159523] [ 8212.164897] ? show_regs.cold.9+0x22/0x2f [ 8212.169862] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8212.173653] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8212.178340] ? watchdog+0x40/0x40 [ 8212.181684] ? hrtimer_interrupt+0x128/0x2c0 [ 8212.184388] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8212.189333] ? apic_timer_interrupt+0xf/0x20 [ 8212.191389] [ 8212.194407] ? native_safe_halt+0x2e/0x50 [ 8212.198836] kvm_wait+0xcd/0xe0 [ 8212.200586] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8212.203267] do_raw_spin_lock+0x177/0x190 [ 8212.205385] _raw_spin_lock+0x21/0x30 [ 8212.207255] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8212.210577] ? do_raw_spin_unlock+0x75/0x190 [ 8212.213744] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8212.220698] ? do_wait_intr+0xf0/0xf0 [ 8212.222017] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8212.226263] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8212.229430] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8212.232713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8212.237196] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8212.239610] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8212.242317] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8212.244952] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8212.247736] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8212.251590] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8212.254636] lmv_intent_open+0x338/0x1150 [lmv] [ 8212.258665] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8212.262519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8212.266963] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8212.268911] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8212.272839] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8212.275837] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8212.279027] ? do_raw_spin_unlock+0x75/0x190 [ 8212.281438] ? _raw_spin_unlock+0x12/0x30 [ 8212.285595] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8212.289389] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8212.291916] ll_atomic_open+0x36a/0xd70 [lustre] [ 8212.295226] lookup_open+0x356/0xb40 [ 8212.297280] do_last+0x313/0xfc0 [ 8212.299484] ? nd_jump_root+0xe5/0x160 [ 8212.301952] ? path_init+0x437/0x520 [ 8212.304506] path_openat+0xf7/0x500 [ 8212.306564] ? do_raw_spin_unlock+0x75/0x190 [ 8212.312226] do_filp_open+0x99/0x140 [ 8212.314645] ? getname_flags+0x6e/0x330 [ 8212.316614] ? __check_object_size+0xff/0x256 [ 8212.319510] ? do_raw_spin_unlock+0x75/0x190 [ 8212.321425] ? _raw_spin_unlock+0x12/0x30 [ 8212.323444] do_sys_openat2+0x2b4/0x410 [ 8212.326107] do_sys_open+0x73/0xa0 [ 8212.328227] __x64_sys_openat+0x24/0x30 [ 8212.330352] do_syscall_64+0xc1/0x440 [ 8212.332562] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8212.335929] RIP: 0033:0x7fdc0afbf332 [ 8212.339659] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8212.353442] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8212.356941] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8212.360015] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8212.363267] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8212.366593] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8212.370008] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8216.057238] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8216.061282] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8216.096443] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8216.105756] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8216.109581] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8216.115935] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8216.131147] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8216.140757] RAX: 0000000000006a0e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8216.147300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8216.151356] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8216.154936] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8216.158027] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8216.164218] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8216.168794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8216.174219] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8216.180489] Call Trace: [ 8216.181388] [ 8216.185646] ? show_regs.cold.9+0x22/0x2f [ 8216.187889] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8216.190661] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8216.192505] ? watchdog+0x40/0x40 [ 8216.194140] ? hrtimer_interrupt+0x128/0x2c0 [ 8216.195931] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8216.200399] ? apic_timer_interrupt+0xf/0x20 [ 8216.202186] [ 8216.203114] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8216.207332] do_raw_spin_lock+0x177/0x190 [ 8216.208788] _raw_spin_lock+0x21/0x30 [ 8216.210511] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8216.216634] ? do_raw_spin_unlock+0x75/0x190 [ 8216.222004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8216.224711] ? do_wait_intr+0xf0/0xf0 [ 8216.226794] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8216.232295] mdc_close+0x53b/0xd70 [mdc] [ 8216.234810] lmv_close+0x18d/0x490 [lmv] [ 8216.239533] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8216.245122] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8216.248383] ll_md_close+0x274/0x800 [lustre] [ 8216.250788] ll_file_release+0x128/0xc50 [lustre] [ 8216.253492] __fput+0x10d/0x440 [ 8216.256168] ____fput+0x12/0x20 [ 8216.260396] task_work_run+0xc6/0x110 [ 8216.262747] exit_to_usermode_loop+0x1dd/0x1f0 [ 8216.267533] do_syscall_64+0x3d6/0x440 [ 8216.269935] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8216.272067] RIP: 0033:0x7fdc0afbfcd8 [ 8216.273843] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8216.290446] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8216.296938] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8216.303220] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8216.308463] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8216.311729] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8216.316020] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8217.055039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6192s! [ 8217.058224] Showing busy workqueues and worker pools: [ 8217.059855] workqueue events: flags=0x0 [ 8217.061225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8217.063813] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8217.067124] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8217.070015] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8217.074266] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8217.076574] pending: kfree_rcu_monitor [ 8217.078137] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8217.080279] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8217.084554] workqueue events_long: flags=0x0 [ 8217.092150] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8217.100438] pending: lu_objects_flush [obdclass] [ 8217.102819] workqueue events_unbound: flags=0x2 [ 8217.105016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8217.107720] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8217.119229] workqueue events_power_efficient: flags=0x80 [ 8217.125363] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8217.131303] pending: neigh_periodic_work, check_lifetime [ 8217.136147] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8217.140772] pending: do_cache_clean, neigh_periodic_work [ 8217.143231] workqueue mm_percpu_wq: flags=0x8 [ 8217.145183] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8217.149930] pending: lru_add_drain_per_cpu [ 8217.153060] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8217.159334] pending: vmstat_update, lru_add_drain_per_cpu [ 8217.162132] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8217.164755] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8217.168045] workqueue writeback: flags=0x4a [ 8217.169476] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8217.174713] pending: wb_workfn, wb_workfn [ 8217.176696] workqueue xprtiod: flags=0xa [ 8217.179870] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8217.185645] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8217.195095] workqueue ipv6_addrconf: flags=0x40008 [ 8217.198735] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8217.202541] pending: addrconf_verify_work [ 8217.204964] workqueue knbd0-recv: flags=0x1a [ 8217.207957] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8217.212340] in-flight: 655:recv_work [ 8217.214689] workqueue ptlrpc_pinger: flags=0x2 [ 8217.219213] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8217.222320] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8217.227853] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6192s workers=4 idle: 36396 38 11188 [ 8217.231352] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8207s workers=2 idle: 658 [ 8228.055818] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8228.060948] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8228.091964] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8228.097895] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8228.102546] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8228.107236] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8228.111164] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8228.127178] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8228.130802] RAX: 0000000000000189 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8228.142668] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8228.146240] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8228.150762] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8228.153827] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8228.157458] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8228.162617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8228.165040] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8228.168196] Call Trace: [ 8228.170886] [ 8228.171388] ? show_regs.cold.9+0x22/0x2f [ 8228.172370] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8228.175469] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8228.176760] ? watchdog+0x40/0x40 [ 8228.177684] ? hrtimer_interrupt+0x128/0x2c0 [ 8228.183219] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8228.186844] ? apic_timer_interrupt+0xf/0x20 [ 8228.189782] [ 8228.194209] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8228.196750] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8228.201219] do_raw_spin_lock+0x177/0x190 [ 8228.203341] _raw_spin_lock+0x21/0x30 [ 8228.205287] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8228.207792] process_one_work+0x2c8/0x700 [ 8228.209352] worker_thread+0x296/0x6e0 [ 8228.210482] ? rescuer_thread+0x570/0x570 [ 8228.211512] kthread+0x1d1/0x200 [ 8228.212389] ? set_kthread_struct+0x70/0x70 [ 8228.213791] ret_from_fork+0x1f/0x30 [ 8240.052666] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 8240.056150] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8240.082379] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8240.087323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8240.091377] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8240.093691] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8240.110617] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8240.117761] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8240.122193] RDX: 0000000000004265 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8240.132969] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8240.138868] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8240.148450] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8240.156431] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8240.167842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8240.171608] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8240.174139] Call Trace: [ 8240.185501] [ 8240.186895] ? show_regs.cold.9+0x22/0x2f [ 8240.193067] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8240.195276] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8240.201025] ? watchdog+0x40/0x40 [ 8240.205419] ? hrtimer_interrupt+0x128/0x2c0 [ 8240.208816] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8240.211987] ? apic_timer_interrupt+0xf/0x20 [ 8240.215003] [ 8240.217289] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8240.221046] do_raw_spin_lock+0x177/0x190 [ 8240.222942] _raw_spin_lock+0x21/0x30 [ 8240.225509] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8240.239112] ? do_raw_spin_unlock+0x75/0x190 [ 8240.243033] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8240.246569] ? do_wait_intr+0xf0/0xf0 [ 8240.252465] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8240.260955] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8240.264966] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8240.270088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8240.275849] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8240.280855] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8240.289482] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8240.296986] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8240.301542] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8240.305954] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8240.308776] lmv_intent_open+0x338/0x1150 [lmv] [ 8240.310766] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8240.318965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8240.331469] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8240.332807] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8240.337035] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8240.347076] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8240.350262] ? do_raw_spin_unlock+0x75/0x190 [ 8240.354433] ? _raw_spin_unlock+0x12/0x30 [ 8240.359920] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8240.362542] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8240.368067] ll_atomic_open+0x36a/0xd70 [lustre] [ 8240.371546] lookup_open+0x356/0xb40 [ 8240.376203] do_last+0x313/0xfc0 [ 8240.379102] ? nd_jump_root+0xe5/0x160 [ 8240.383635] ? path_init+0x437/0x520 [ 8240.390056] path_openat+0xf7/0x500 [ 8240.393959] ? do_raw_spin_unlock+0x75/0x190 [ 8240.404267] do_filp_open+0x99/0x140 [ 8240.409510] ? getname_flags+0x6e/0x330 [ 8240.413909] ? __check_object_size+0xff/0x256 [ 8240.418955] ? do_raw_spin_unlock+0x75/0x190 [ 8240.421464] ? _raw_spin_unlock+0x12/0x30 [ 8240.427367] do_sys_openat2+0x2b4/0x410 [ 8240.428930] do_sys_open+0x73/0xa0 [ 8240.431936] __x64_sys_openat+0x24/0x30 [ 8240.434137] do_syscall_64+0xc1/0x440 [ 8240.440902] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8240.444415] RIP: 0033:0x7fdc0afbf332 [ 8240.453125] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8240.471142] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8240.483866] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8240.488959] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8240.506716] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8240.512850] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8240.522306] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8244.048012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8244.053183] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8244.055594] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8244.081999] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8244.082014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8244.082017] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8244.082033] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8244.082037] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8244.082043] RAX: 0000000000001107 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8244.082045] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8244.082048] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8244.082050] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8244.082052] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8244.082125] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8244.082133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8244.082135] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8244.082142] Call Trace: [ 8244.082148] [ 8244.082154] ? show_regs.cold.9+0x22/0x2f [ 8244.082168] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8244.082175] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8244.082179] ? watchdog+0x40/0x40 [ 8244.082183] ? hrtimer_interrupt+0x128/0x2c0 [ 8244.082187] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8244.082199] ? apic_timer_interrupt+0xf/0x20 [ 8244.082204] [ 8244.082207] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8244.082212] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8244.082220] ? dump_stack+0xf6/0x10e [ 8244.082227] do_raw_spin_lock+0x177/0x190 [ 8244.082232] _raw_spin_lock+0x21/0x30 [ 8244.082239] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8244.082793] obd_mod_free+0x46/0x140 [ptlrpc] [ 8244.083085] mdc_commit_open+0x91/0xb0 [mdc] [ 8244.083264] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8244.083494] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8244.083990] after_reply+0xe2c/0x1780 [ptlrpc] [ 8244.084462] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8244.084747] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8244.085402] ? do_wait_intr+0xf0/0xf0 [ 8244.085414] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8244.085662] kthread+0x1d1/0x200 [ 8244.085670] ? set_kthread_struct+0x70/0x70 [ 8244.085673] ret_from_fork+0x1f/0x30 [ 8244.362989] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8244.397989] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8244.403586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8244.413408] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8244.421878] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8244.437229] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8244.442413] RAX: 0000000000007b30 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8244.445882] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8244.452442] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8244.456879] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8244.463513] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8244.469065] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8244.474764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8244.480255] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8244.486916] Call Trace: [ 8244.487831] [ 8244.488545] ? show_regs.cold.9+0x22/0x2f [ 8244.489911] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8244.502839] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8244.509854] ? watchdog+0x40/0x40 [ 8244.512812] ? hrtimer_interrupt+0x128/0x2c0 [ 8244.516082] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8244.521226] ? apic_timer_interrupt+0xf/0x20 [ 8244.526358] [ 8244.527070] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8244.532406] do_raw_spin_lock+0x177/0x190 [ 8244.536473] _raw_spin_lock+0x21/0x30 [ 8244.542047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8244.547053] ? do_raw_spin_unlock+0x75/0x190 [ 8244.551760] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8244.554272] ? do_wait_intr+0xf0/0xf0 [ 8244.555988] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8244.559918] mdc_close+0x53b/0xd70 [mdc] [ 8244.561776] lmv_close+0x18d/0x490 [lmv] [ 8244.563323] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8244.567827] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8244.577468] ll_md_close+0x274/0x800 [lustre] [ 8244.580350] ll_file_release+0x128/0xc50 [lustre] [ 8244.587133] __fput+0x10d/0x440 [ 8244.589204] ____fput+0x12/0x20 [ 8244.594157] task_work_run+0xc6/0x110 [ 8244.600046] exit_to_usermode_loop+0x1dd/0x1f0 [ 8244.604722] do_syscall_64+0x3d6/0x440 [ 8244.610848] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8244.614714] RIP: 0033:0x7fdc0afbfcd8 [ 8244.616433] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8244.623844] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8244.634568] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8244.641092] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8244.655140] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8244.658498] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8244.661960] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8247.276550] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6222s! [ 8247.290257] Showing busy workqueues and worker pools: [ 8247.297774] workqueue events: flags=0x0 [ 8247.301875] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8247.306537] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8247.311493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8247.319945] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8247.333597] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8247.335455] pending: kfree_rcu_monitor [ 8247.341867] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8247.347042] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8247.353506] workqueue events_long: flags=0x0 [ 8247.356913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8247.370649] pending: lu_objects_flush [obdclass] [ 8247.377225] workqueue events_unbound: flags=0x2 [ 8247.381594] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8247.388643] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8247.396163] workqueue events_power_efficient: flags=0x80 [ 8247.402001] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8247.409555] pending: neigh_periodic_work, check_lifetime [ 8247.411439] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8247.423424] pending: do_cache_clean, neigh_periodic_work [ 8247.429725] workqueue mm_percpu_wq: flags=0x8 [ 8247.435518] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8247.438651] pending: lru_add_drain_per_cpu [ 8247.445719] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8247.454747] pending: vmstat_update, lru_add_drain_per_cpu [ 8247.458478] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8247.463444] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8247.465218] workqueue writeback: flags=0x4a [ 8247.476956] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8247.488282] pending: wb_workfn, wb_workfn [ 8247.490573] workqueue xprtiod: flags=0xa [ 8247.498596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8247.505993] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8247.518533] workqueue ipv6_addrconf: flags=0x40008 [ 8247.523604] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8247.531071] pending: addrconf_verify_work [ 8247.535490] workqueue knbd0-recv: flags=0x1a [ 8247.537534] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8247.548114] in-flight: 655:recv_work [ 8247.558937] workqueue ptlrpc_pinger: flags=0x2 [ 8247.561923] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8247.567940] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8247.574488] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6222s workers=4 idle: 36396 38 11188 [ 8247.580324] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8238s workers=2 idle: 658 [ 8256.051025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 8256.059821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8256.130759] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8256.147811] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8256.159564] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8256.170623] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8256.173669] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8256.182914] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8256.192180] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8256.200821] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8256.203909] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8256.207672] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8256.212575] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8256.219135] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8256.230494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8256.235771] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8256.240671] Call Trace: [ 8256.244299] [ 8256.244857] ? show_regs.cold.9+0x22/0x2f [ 8256.255473] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8256.259479] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8256.264484] ? watchdog+0x40/0x40 [ 8256.267126] ? hrtimer_interrupt+0x128/0x2c0 [ 8256.271579] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8256.277565] ? apic_timer_interrupt+0xf/0x20 [ 8256.286439] [ 8256.288128] ? native_safe_halt+0x2e/0x50 [ 8256.293557] kvm_wait+0xcd/0xe0 [ 8256.296257] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8256.300522] do_raw_spin_lock+0x177/0x190 [ 8256.306582] _raw_spin_lock+0x21/0x30 [ 8256.311583] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8256.322550] process_one_work+0x2c8/0x700 [ 8256.324086] worker_thread+0x296/0x6e0 [ 8256.332102] ? rescuer_thread+0x570/0x570 [ 8256.337182] kthread+0x1d1/0x200 [ 8256.338149] ? set_kthread_struct+0x70/0x70 [ 8256.340256] ret_from_fork+0x1f/0x30 [ 8268.052063] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 8268.055524] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8268.080214] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8268.090988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8268.099209] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8268.104115] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8268.120652] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8268.124645] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8268.131869] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8268.136094] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8268.141114] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8268.143450] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8268.156817] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8268.166301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8268.169892] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8268.176116] Call Trace: [ 8268.178854] [ 8268.179800] ? show_regs.cold.9+0x22/0x2f [ 8268.182403] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8268.191334] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8268.196480] ? watchdog+0x40/0x40 [ 8268.198297] ? hrtimer_interrupt+0x128/0x2c0 [ 8268.200846] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8268.203887] ? apic_timer_interrupt+0xf/0x20 [ 8268.206266] [ 8268.207452] ? native_safe_halt+0x2e/0x50 [ 8268.210329] kvm_wait+0xcd/0xe0 [ 8268.212257] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8268.222128] do_raw_spin_lock+0x177/0x190 [ 8268.225587] _raw_spin_lock+0x21/0x30 [ 8268.232474] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8268.242428] ? do_raw_spin_unlock+0x75/0x190 [ 8268.247739] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8268.252062] ? do_wait_intr+0xf0/0xf0 [ 8268.260897] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8268.265144] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8268.268157] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8268.279605] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8268.286279] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8268.293307] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8268.298698] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8268.300745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8268.307253] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8268.310485] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8268.313474] lmv_intent_open+0x338/0x1150 [lmv] [ 8268.316068] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8268.321221] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8268.327061] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8268.336735] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8268.343095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8268.347381] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8268.348946] ? do_raw_spin_unlock+0x75/0x190 [ 8268.350341] ? _raw_spin_unlock+0x12/0x30 [ 8268.353954] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8268.355853] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8268.359939] ll_atomic_open+0x36a/0xd70 [lustre] [ 8268.362523] lookup_open+0x356/0xb40 [ 8268.363903] do_last+0x313/0xfc0 [ 8268.365866] ? nd_jump_root+0xe5/0x160 [ 8268.367287] ? path_init+0x437/0x520 [ 8268.368974] path_openat+0xf7/0x500 [ 8268.371018] ? do_raw_spin_unlock+0x75/0x190 [ 8268.372674] do_filp_open+0x99/0x140 [ 8268.376795] ? getname_flags+0x6e/0x330 [ 8268.379003] ? __check_object_size+0xff/0x256 [ 8268.380569] ? do_raw_spin_unlock+0x75/0x190 [ 8268.383312] ? _raw_spin_unlock+0x12/0x30 [ 8268.385525] do_sys_openat2+0x2b4/0x410 [ 8268.387760] do_sys_open+0x73/0xa0 [ 8268.394128] __x64_sys_openat+0x24/0x30 [ 8268.401405] do_syscall_64+0xc1/0x440 [ 8268.403515] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8268.406027] RIP: 0033:0x7fdc0afbf332 [ 8268.409626] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8268.423182] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8268.433236] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8268.444668] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8268.449387] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8268.453320] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8268.457390] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8272.050025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8272.054066] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8272.058759] Modules linked in: [ 8272.061579] Modules linked in: [ 8272.061582] lustre(O) [ 8272.062869] lustre(O) [ 8272.064361] osp(O) [ 8272.065235] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8272.066845] ofd(O) [ 8272.067798] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8272.067806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8272.067808] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8272.067822] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8272.067825] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8272.067830] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8272.067832] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8272.067835] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8272.067837] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8272.067839] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8272.067843] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8272.067846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8272.067848] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8272.067855] Call Trace: [ 8272.067860] [ 8272.067865] ? show_regs.cold.9+0x22/0x2f [ 8272.067875] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8272.096380] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8272.096446] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8272.096454] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8272.096458] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8272.096471] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8272.096475] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8272.096480] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8272.096482] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8272.096484] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8272.096486] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8272.096489] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8272.096492] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8272.096495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8272.096497] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8272.096506] Call Trace: [ 8272.096511] [ 8272.096517] ? show_regs.cold.9+0x22/0x2f [ 8272.096527] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8272.096534] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8272.096538] ? watchdog+0x40/0x40 [ 8272.096541] ? hrtimer_interrupt+0x128/0x2c0 [ 8272.096546] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8272.096552] ? apic_timer_interrupt+0xf/0x20 [ 8272.096556] [ 8272.096559] ? native_safe_halt+0x2e/0x50 [ 8272.096562] kvm_wait+0xcd/0xe0 [ 8272.096568] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8272.096575] ? dump_stack+0xf6/0x10e [ 8272.096580] do_raw_spin_lock+0x177/0x190 [ 8272.096584] _raw_spin_lock+0x21/0x30 [ 8272.096589] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8272.097598] obd_mod_free+0x46/0x140 [ptlrpc] [ 8272.098239] mdc_commit_open+0x91/0xb0 [mdc] [ 8272.098278] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8272.102523] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8272.102540] ? watchdog+0x40/0x40 [ 8272.102547] ? hrtimer_interrupt+0x128/0x2c0 [ 8272.102551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8272.102559] ? apic_timer_interrupt+0xf/0x20 [ 8272.102564] [ 8272.102568] ? native_safe_halt+0x2e/0x50 [ 8272.102572] kvm_wait+0xcd/0xe0 [ 8272.102580] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8272.102590] do_raw_spin_lock+0x177/0x190 [ 8272.102594] _raw_spin_lock+0x21/0x30 [ 8272.102598] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8272.102902] ? do_raw_spin_unlock+0x75/0x190 [ 8272.102908] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8272.103211] ? do_wait_intr+0xf0/0xf0 [ 8272.103217] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8272.103398] mdc_close+0x53b/0xd70 [mdc] [ 8272.103436] lmv_close+0x18d/0x490 [lmv] [ 8272.103461] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8272.103569] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8272.105942] ll_md_close+0x274/0x800 [lustre] [ 8272.106194] ll_file_release+0x128/0xc50 [lustre] [ 8272.107085] __fput+0x10d/0x440 [ 8272.107157] ____fput+0x12/0x20 [ 8272.107162] task_work_run+0xc6/0x110 [ 8272.107170] exit_to_usermode_loop+0x1dd/0x1f0 [ 8272.107178] do_syscall_64+0x3d6/0x440 [ 8272.107182] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8272.107190] RIP: 0033:0x7fdc0afbfcd8 [ 8272.107197] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8272.107201] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8272.107205] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8272.107208] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8272.107210] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8272.107213] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8272.107215] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8272.483775] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8272.494229] after_reply+0xe2c/0x1780 [ptlrpc] [ 8272.496246] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8272.504359] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8272.506562] ? do_wait_intr+0xf0/0xf0 [ 8272.514105] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8272.519905] kthread+0x1d1/0x200 [ 8272.524354] ? set_kthread_struct+0x70/0x70 [ 8272.530827] ret_from_fork+0x1f/0x30 [ 8277.983081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6253s! [ 8277.988302] Showing busy workqueues and worker pools: [ 8277.990622] workqueue events: flags=0x0 [ 8277.992200] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8277.994895] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8277.998046] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8278.000288] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8278.005459] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8278.009265] pending: kfree_rcu_monitor [ 8278.011467] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8278.014938] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8278.024782] workqueue events_long: flags=0x0 [ 8278.028875] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8278.031763] pending: lu_objects_flush [obdclass] [ 8278.034201] workqueue events_unbound: flags=0x2 [ 8278.036210] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8278.040729] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8278.044774] workqueue events_power_efficient: flags=0x80 [ 8278.047341] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8278.050746] pending: neigh_periodic_work, check_lifetime [ 8278.053974] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8278.057250] pending: do_cache_clean, neigh_periodic_work [ 8278.059294] workqueue mm_percpu_wq: flags=0x8 [ 8278.061534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8278.065229] pending: lru_add_drain_per_cpu [ 8278.067486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8278.073094] pending: vmstat_update, lru_add_drain_per_cpu [ 8278.075076] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8278.078147] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8278.081163] workqueue writeback: flags=0x4a [ 8278.083111] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8278.086215] pending: wb_workfn, wb_workfn [ 8278.089373] workqueue xprtiod: flags=0xa [ 8278.091125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8278.095461] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8278.101297] workqueue ipv6_addrconf: flags=0x40008 [ 8278.103339] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8278.106734] pending: addrconf_verify_work [ 8278.109346] workqueue knbd0-recv: flags=0x1a [ 8278.112800] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8278.116834] in-flight: 655:recv_work [ 8278.119565] workqueue ptlrpc_pinger: flags=0x2 [ 8278.122427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8278.125219] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8278.128664] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6253s workers=4 idle: 36396 38 11188 [ 8278.134703] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8268s workers=2 idle: 658 [ 8284.051058] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 8284.054656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8284.079036] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8284.083748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8284.088115] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8284.091641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8284.095040] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8284.103566] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8284.107428] RAX: 0000000000005050 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8284.111325] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8284.114234] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8284.116724] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8284.119756] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8284.122987] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8284.126910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8284.128671] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8284.131232] Call Trace: [ 8284.132290] [ 8284.133710] ? show_regs.cold.9+0x22/0x2f [ 8284.135511] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8284.137299] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8284.138999] ? watchdog+0x40/0x40 [ 8284.140166] ? hrtimer_interrupt+0x128/0x2c0 [ 8284.141774] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8284.143694] ? apic_timer_interrupt+0xf/0x20 [ 8284.145297] [ 8284.146074] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8284.148100] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8284.150046] do_raw_spin_lock+0x177/0x190 [ 8284.151493] _raw_spin_lock+0x21/0x30 [ 8284.152845] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8284.155062] process_one_work+0x2c8/0x700 [ 8284.156798] worker_thread+0x296/0x6e0 [ 8284.158110] ? rescuer_thread+0x570/0x570 [ 8284.159738] kthread+0x1d1/0x200 [ 8284.160779] ? set_kthread_struct+0x70/0x70 [ 8284.162366] ret_from_fork+0x1f/0x30 [ 8296.053536] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 8296.059689] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8296.089366] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8296.094291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8296.101410] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8296.106020] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8296.117158] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8296.127946] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8296.133515] RDX: 0000000000003f62 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8296.142806] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8296.148114] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8296.152053] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8296.161683] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8296.165333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8296.174112] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8296.182397] Call Trace: [ 8296.183928] [ 8296.186683] ? show_regs.cold.9+0x22/0x2f [ 8296.188925] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8296.191704] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8296.193371] ? watchdog+0x40/0x40 [ 8296.194996] ? hrtimer_interrupt+0x128/0x2c0 [ 8296.196754] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8296.198847] ? apic_timer_interrupt+0xf/0x20 [ 8296.201228] [ 8296.202153] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8296.204862] do_raw_spin_lock+0x177/0x190 [ 8296.206798] _raw_spin_lock+0x21/0x30 [ 8296.209250] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8296.215401] ? do_raw_spin_unlock+0x75/0x190 [ 8296.219382] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8296.229065] ? do_wait_intr+0xf0/0xf0 [ 8296.232020] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8296.236016] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8296.240639] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8296.249491] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8296.254451] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8296.256955] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8296.259403] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8296.261851] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8296.264582] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8296.269602] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8296.274119] lmv_intent_open+0x338/0x1150 [lmv] [ 8296.281969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8296.287423] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8296.292032] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8296.295067] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8296.300214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8296.303007] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8296.305447] ? do_raw_spin_unlock+0x75/0x190 [ 8296.308265] ? _raw_spin_unlock+0x12/0x30 [ 8296.309757] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8296.312525] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8296.315380] ll_atomic_open+0x36a/0xd70 [lustre] [ 8296.317529] lookup_open+0x356/0xb40 [ 8296.319596] do_last+0x313/0xfc0 [ 8296.321052] ? nd_jump_root+0xe5/0x160 [ 8296.323019] ? path_init+0x437/0x520 [ 8296.324478] path_openat+0xf7/0x500 [ 8296.327952] ? do_raw_spin_unlock+0x75/0x190 [ 8296.332321] do_filp_open+0x99/0x140 [ 8296.333738] ? getname_flags+0x6e/0x330 [ 8296.336540] ? __check_object_size+0xff/0x256 [ 8296.338731] ? do_raw_spin_unlock+0x75/0x190 [ 8296.342165] ? _raw_spin_unlock+0x12/0x30 [ 8296.344760] do_sys_openat2+0x2b4/0x410 [ 8296.347054] do_sys_open+0x73/0xa0 [ 8296.348045] __x64_sys_openat+0x24/0x30 [ 8296.349460] do_syscall_64+0xc1/0x440 [ 8296.350733] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8296.352728] RIP: 0033:0x7fdc0afbf332 [ 8296.353772] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8296.363910] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8296.375265] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8296.379543] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8296.384008] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8296.388577] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8296.393667] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8300.054051] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 8300.057948] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8300.083288] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8300.087631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8300.091459] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8300.093218] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8300.101495] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8300.104983] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8300.107863] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8300.110964] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8300.114019] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8300.116949] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8300.119517] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8300.122948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8300.125488] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8300.128309] Call Trace: [ 8300.129295] [ 8300.130047] ? show_regs.cold.9+0x22/0x2f [ 8300.131967] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8300.134010] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8300.135660] ? watchdog+0x40/0x40 [ 8300.137317] ? hrtimer_interrupt+0x128/0x2c0 [ 8300.139052] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8300.141208] ? apic_timer_interrupt+0xf/0x20 [ 8300.143404] [ 8300.144354] ? native_safe_halt+0x2e/0x50 [ 8300.146064] kvm_wait+0xcd/0xe0 [ 8300.147400] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8300.149617] do_raw_spin_lock+0x177/0x190 [ 8300.151525] _raw_spin_lock+0x21/0x30 [ 8300.153387] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8300.155750] ? do_raw_spin_unlock+0x75/0x190 [ 8300.157962] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8300.160365] ? do_wait_intr+0xf0/0xf0 [ 8300.162181] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8300.164577] mdc_close+0x53b/0xd70 [mdc] [ 8300.166245] lmv_close+0x18d/0x490 [lmv] [ 8300.168037] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8300.170434] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8300.172232] ll_md_close+0x274/0x800 [lustre] [ 8300.173971] ll_file_release+0x128/0xc50 [lustre] [ 8300.176045] __fput+0x10d/0x440 [ 8300.177669] ____fput+0x12/0x20 [ 8300.179186] task_work_run+0xc6/0x110 [ 8300.181239] exit_to_usermode_loop+0x1dd/0x1f0 [ 8300.183982] do_syscall_64+0x3d6/0x440 [ 8300.185353] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8300.188191] RIP: 0033:0x7fdc0afbfcd8 [ 8300.190267] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8300.200633] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8300.204084] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8300.208568] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8300.214221] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8300.218232] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8300.223659] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8304.048195] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8304.053320] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8304.072324] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8304.079219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8304.081626] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8304.084450] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8304.089885] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8304.093721] RAX: 0000000000007e24 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8304.096945] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8304.099314] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8304.101789] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8304.103843] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8304.108775] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8304.113242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8304.117013] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8304.121007] Call Trace: [ 8304.122383] [ 8304.123281] ? show_regs.cold.9+0x22/0x2f [ 8304.125528] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8304.127493] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8304.130110] ? watchdog+0x40/0x40 [ 8304.131559] ? hrtimer_interrupt+0x128/0x2c0 [ 8304.133497] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8304.147405] ? apic_timer_interrupt+0xf/0x20 [ 8304.150402] [ 8304.151413] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8304.153370] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8304.160103] ? dump_stack+0xf6/0x10e [ 8304.161448] do_raw_spin_lock+0x177/0x190 [ 8304.164746] _raw_spin_lock+0x21/0x30 [ 8304.167425] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8304.170382] obd_mod_free+0x46/0x140 [ptlrpc] [ 8304.173425] mdc_commit_open+0x91/0xb0 [mdc] [ 8304.178533] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8304.189656] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8304.197698] after_reply+0xe2c/0x1780 [ptlrpc] [ 8304.208374] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8304.213959] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8304.221630] ? do_wait_intr+0xf0/0xf0 [ 8304.224788] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8304.228412] kthread+0x1d1/0x200 [ 8304.232244] ? set_kthread_struct+0x70/0x70 [ 8304.236987] ret_from_fork+0x1f/0x30 [ 8308.191038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6283s! [ 8308.198696] Showing busy workqueues and worker pools: [ 8308.204917] workqueue events: flags=0x0 [ 8308.206298] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8308.224943] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8308.231671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8308.236746] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8308.248614] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8308.251461] pending: kfree_rcu_monitor [ 8308.253604] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8308.256644] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8308.269570] workqueue events_long: flags=0x0 [ 8308.273091] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8308.276498] pending: lu_objects_flush [obdclass] [ 8308.281985] workqueue events_unbound: flags=0x2 [ 8308.284434] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8308.289917] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8308.298199] workqueue events_power_efficient: flags=0x80 [ 8308.301938] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8308.307095] pending: neigh_periodic_work, check_lifetime [ 8308.309970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8308.312218] pending: do_cache_clean, neigh_periodic_work [ 8308.318827] workqueue mm_percpu_wq: flags=0x8 [ 8308.323160] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8308.327322] pending: lru_add_drain_per_cpu [ 8308.331021] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8308.334620] pending: vmstat_update, lru_add_drain_per_cpu [ 8308.336857] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8308.347390] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8308.352256] workqueue writeback: flags=0x4a [ 8308.360445] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8308.366987] pending: wb_workfn, wb_workfn [ 8308.373385] workqueue xprtiod: flags=0xa [ 8308.375985] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8308.379961] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8308.385848] workqueue ipv6_addrconf: flags=0x40008 [ 8308.390283] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8308.397699] pending: addrconf_verify_work [ 8308.402953] workqueue knbd0-recv: flags=0x1a [ 8308.408119] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8308.412171] in-flight: 655:recv_work [ 8308.415818] workqueue ptlrpc_pinger: flags=0x2 [ 8308.417885] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8308.426760] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8308.430853] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6283s workers=4 idle: 36396 38 11188 [ 8308.436492] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8298s workers=2 idle: 658 [ 8312.053358] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 8312.056720] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8312.086917] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8312.092761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8312.100372] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8312.103901] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8312.107317] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8312.118679] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8312.124272] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8312.126916] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8312.133161] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8312.139046] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8312.145656] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8312.149506] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8312.154242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8312.156953] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8312.160183] Call Trace: [ 8312.164129] [ 8312.165262] ? show_regs.cold.9+0x22/0x2f [ 8312.168296] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8312.177829] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8312.180112] ? watchdog+0x40/0x40 [ 8312.181747] ? hrtimer_interrupt+0x128/0x2c0 [ 8312.186400] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8312.189244] ? apic_timer_interrupt+0xf/0x20 [ 8312.191547] [ 8312.192686] ? native_safe_halt+0x2e/0x50 [ 8312.195701] kvm_wait+0xcd/0xe0 [ 8312.201024] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8312.204033] do_raw_spin_lock+0x177/0x190 [ 8312.210529] _raw_spin_lock+0x21/0x30 [ 8312.212167] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8312.217454] process_one_work+0x2c8/0x700 [ 8312.221330] worker_thread+0x296/0x6e0 [ 8312.223183] ? rescuer_thread+0x570/0x570 [ 8312.225180] kthread+0x1d1/0x200 [ 8312.226300] ? set_kthread_struct+0x70/0x70 [ 8312.231198] ret_from_fork+0x1f/0x30 [ 8324.052577] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8324.060625] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8324.091810] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8324.104566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8324.111555] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8324.125740] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8324.146589] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8324.158447] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8324.164829] RDX: 000000000000169f RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8324.176379] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8324.178518] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8324.187421] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8324.193617] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8324.197967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8324.206394] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8324.217195] Call Trace: [ 8324.221530] [ 8324.223520] ? show_regs.cold.9+0x22/0x2f [ 8324.228340] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8324.233562] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8324.236714] ? watchdog+0x40/0x40 [ 8324.238995] ? hrtimer_interrupt+0x128/0x2c0 [ 8324.241190] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8324.242613] ? apic_timer_interrupt+0xf/0x20 [ 8324.244337] [ 8324.244914] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8324.251317] do_raw_spin_lock+0x177/0x190 [ 8324.252751] _raw_spin_lock+0x21/0x30 [ 8324.254072] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8324.264615] ? do_raw_spin_unlock+0x75/0x190 [ 8324.267998] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8324.275351] ? do_wait_intr+0xf0/0xf0 [ 8324.277580] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8324.285353] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8324.291602] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8324.296954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8324.299762] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8324.301806] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8324.305438] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8324.307479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8324.322018] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8324.328439] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8324.331923] lmv_intent_open+0x338/0x1150 [lmv] [ 8324.335415] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8324.337942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8324.339275] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8324.348309] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8324.350498] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8324.359849] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8324.367137] ? do_raw_spin_unlock+0x75/0x190 [ 8324.370646] ? _raw_spin_unlock+0x12/0x30 [ 8324.372282] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8324.374176] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8324.376286] ll_atomic_open+0x36a/0xd70 [lustre] [ 8324.383877] lookup_open+0x356/0xb40 [ 8324.385529] do_last+0x313/0xfc0 [ 8324.387116] ? nd_jump_root+0xe5/0x160 [ 8324.388377] ? path_init+0x437/0x520 [ 8324.390008] path_openat+0xf7/0x500 [ 8324.391505] ? do_raw_spin_unlock+0x75/0x190 [ 8324.394980] do_filp_open+0x99/0x140 [ 8324.399836] ? getname_flags+0x6e/0x330 [ 8324.402250] ? __check_object_size+0xff/0x256 [ 8324.405314] ? do_raw_spin_unlock+0x75/0x190 [ 8324.409312] ? _raw_spin_unlock+0x12/0x30 [ 8324.411562] do_sys_openat2+0x2b4/0x410 [ 8324.418032] do_sys_open+0x73/0xa0 [ 8324.420840] __x64_sys_openat+0x24/0x30 [ 8324.424432] do_syscall_64+0xc1/0x440 [ 8324.426810] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8324.429966] RIP: 0033:0x7fdc0afbf332 [ 8324.431267] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8324.444834] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8324.446774] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8324.466668] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8324.473576] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8324.486437] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8324.491005] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8328.054023] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 8328.057985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8328.084803] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8328.090793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8328.094566] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.099292] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8328.111003] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8328.115148] RAX: 0000000000005004 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8328.120902] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8328.123727] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8328.126782] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8328.130236] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8328.134901] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8328.141891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8328.147474] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8328.154340] Call Trace: [ 8328.155835] [ 8328.159976] ? show_regs.cold.9+0x22/0x2f [ 8328.162537] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8328.168598] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8328.176971] ? watchdog+0x40/0x40 [ 8328.180569] ? hrtimer_interrupt+0x128/0x2c0 [ 8328.184220] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8328.186839] ? apic_timer_interrupt+0xf/0x20 [ 8328.189154] [ 8328.190309] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.192982] do_raw_spin_lock+0x177/0x190 [ 8328.195592] _raw_spin_lock+0x21/0x30 [ 8328.197863] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8328.200744] ? do_raw_spin_unlock+0x75/0x190 [ 8328.202575] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8328.208678] ? do_wait_intr+0xf0/0xf0 [ 8328.210243] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8328.216115] mdc_close+0x53b/0xd70 [mdc] [ 8328.217838] lmv_close+0x18d/0x490 [lmv] [ 8328.223384] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8328.228128] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8328.230800] ll_md_close+0x274/0x800 [lustre] [ 8328.234062] ll_file_release+0x128/0xc50 [lustre] [ 8328.239741] __fput+0x10d/0x440 [ 8328.243568] ____fput+0x12/0x20 [ 8328.247536] task_work_run+0xc6/0x110 [ 8328.250244] exit_to_usermode_loop+0x1dd/0x1f0 [ 8328.255982] do_syscall_64+0x3d6/0x440 [ 8328.259048] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8328.264122] RIP: 0033:0x7fdc0afbfcd8 [ 8328.265847] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8328.285452] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8328.294072] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8328.304365] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8328.309212] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8328.313135] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8328.323009] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8336.049889] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8336.055053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8336.110841] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8336.123737] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8336.129780] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8336.133469] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8336.147117] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8336.150417] RAX: 0000000000007649 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8336.152899] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8336.155717] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8336.159114] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8336.162256] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8336.165363] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8336.168901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8336.171274] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8336.174328] Call Trace: [ 8336.175444] [ 8336.176399] ? show_regs.cold.9+0x22/0x2f [ 8336.178443] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8336.180224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8336.181815] ? watchdog+0x40/0x40 [ 8336.182737] ? hrtimer_interrupt+0x128/0x2c0 [ 8336.185491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8336.187471] ? apic_timer_interrupt+0xf/0x20 [ 8336.195149] [ 8336.195688] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8336.200311] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8336.202670] ? dump_stack+0xf6/0x10e [ 8336.204733] do_raw_spin_lock+0x177/0x190 [ 8336.210748] _raw_spin_lock+0x21/0x30 [ 8336.214115] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8336.220579] obd_mod_free+0x46/0x140 [ptlrpc] [ 8336.224364] mdc_commit_open+0x91/0xb0 [mdc] [ 8336.233534] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8336.242738] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8336.246365] after_reply+0xe2c/0x1780 [ptlrpc] [ 8336.249392] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8336.255184] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8336.257681] ? do_wait_intr+0xf0/0xf0 [ 8336.260178] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8336.268589] kthread+0x1d1/0x200 [ 8336.271835] ? set_kthread_struct+0x70/0x70 [ 8336.280149] ret_from_fork+0x1f/0x30 [ 8338.911142] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6313s! [ 8338.916840] Showing busy workqueues and worker pools: [ 8338.923079] workqueue events: flags=0x0 [ 8338.928605] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8338.931374] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8338.938308] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8338.941897] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8338.954212] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8338.957444] pending: kfree_rcu_monitor [ 8338.959662] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8338.962783] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8338.972408] workqueue events_long: flags=0x0 [ 8338.977295] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8338.982545] pending: lu_objects_flush [obdclass] [ 8338.985712] workqueue events_unbound: flags=0x2 [ 8338.987856] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8338.991430] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8338.996570] workqueue events_power_efficient: flags=0x80 [ 8339.003448] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8339.012295] pending: neigh_periodic_work, check_lifetime [ 8339.016529] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8339.020119] pending: do_cache_clean, neigh_periodic_work [ 8339.023752] workqueue mm_percpu_wq: flags=0x8 [ 8339.026608] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8339.035727] pending: lru_add_drain_per_cpu [ 8339.040842] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8339.045784] pending: vmstat_update, lru_add_drain_per_cpu [ 8339.048138] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8339.051088] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8339.054146] workqueue writeback: flags=0x4a [ 8339.056137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8339.059827] pending: wb_workfn, wb_workfn [ 8339.062685] workqueue xprtiod: flags=0xa [ 8339.064707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8339.068360] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8339.072743] workqueue ipv6_addrconf: flags=0x40008 [ 8339.077559] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8339.082444] pending: addrconf_verify_work [ 8339.086685] workqueue knbd0-recv: flags=0x1a [ 8339.088840] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8339.093304] in-flight: 655:recv_work [ 8339.096141] workqueue ptlrpc_pinger: flags=0x2 [ 8339.098245] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8339.102024] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8339.108510] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6314s workers=4 idle: 36396 38 11188 [ 8339.116779] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8329s workers=2 idle: 658 [ 8340.051035] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 8340.054554] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8340.076102] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8340.081625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8340.086359] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8340.090288] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8340.092594] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8340.102526] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8340.106468] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8340.109969] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8340.113535] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8340.116936] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8340.119563] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8340.123205] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8340.126924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8340.129564] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8340.133356] Call Trace: [ 8340.134804] [ 8340.135711] ? show_regs.cold.9+0x22/0x2f [ 8340.137619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8340.139953] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8340.142359] ? watchdog+0x40/0x40 [ 8340.144003] ? hrtimer_interrupt+0x128/0x2c0 [ 8340.146048] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8340.148532] ? apic_timer_interrupt+0xf/0x20 [ 8340.150110] [ 8340.151153] ? native_safe_halt+0x2e/0x50 [ 8340.153064] kvm_wait+0xcd/0xe0 [ 8340.155000] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8340.157626] do_raw_spin_lock+0x177/0x190 [ 8340.159546] _raw_spin_lock+0x21/0x30 [ 8340.161283] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8340.163938] process_one_work+0x2c8/0x700 [ 8340.165353] worker_thread+0x296/0x6e0 [ 8340.166831] ? rescuer_thread+0x570/0x570 [ 8340.168834] kthread+0x1d1/0x200 [ 8340.170514] ? set_kthread_struct+0x70/0x70 [ 8340.172567] ret_from_fork+0x1f/0x30 [ 8352.052029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8352.055464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8352.073389] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8352.078661] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8352.082955] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8352.085077] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8352.095167] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8352.099447] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8352.103017] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8352.106277] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8352.108542] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8352.112181] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8352.114983] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8352.118328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8352.121189] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8352.124945] Call Trace: [ 8352.126531] [ 8352.127581] ? show_regs.cold.9+0x22/0x2f [ 8352.129440] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8352.131153] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8352.132628] ? watchdog+0x40/0x40 [ 8352.134185] ? hrtimer_interrupt+0x128/0x2c0 [ 8352.135575] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8352.136978] ? apic_timer_interrupt+0xf/0x20 [ 8352.138949] [ 8352.139964] ? native_safe_halt+0x2e/0x50 [ 8352.142473] kvm_wait+0xcd/0xe0 [ 8352.143965] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8352.146865] do_raw_spin_lock+0x177/0x190 [ 8352.148717] _raw_spin_lock+0x21/0x30 [ 8352.150831] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8352.153756] ? do_raw_spin_unlock+0x75/0x190 [ 8352.155318] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8352.158344] ? do_wait_intr+0xf0/0xf0 [ 8352.160182] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8352.163358] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8352.166449] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8352.169769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8352.172773] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8352.176115] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8352.178952] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8352.181212] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8352.183989] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8352.187392] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8352.190447] lmv_intent_open+0x338/0x1150 [lmv] [ 8352.192744] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8352.195763] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8352.198647] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8352.201485] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8352.203802] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8352.206346] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8352.208762] ? do_raw_spin_unlock+0x75/0x190 [ 8352.210851] ? _raw_spin_unlock+0x12/0x30 [ 8352.212981] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8352.215580] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8352.217945] ll_atomic_open+0x36a/0xd70 [lustre] [ 8352.220574] lookup_open+0x356/0xb40 [ 8352.222536] do_last+0x313/0xfc0 [ 8352.224406] ? nd_jump_root+0xe5/0x160 [ 8352.226209] ? path_init+0x437/0x520 [ 8352.228314] path_openat+0xf7/0x500 [ 8352.230442] ? do_raw_spin_unlock+0x75/0x190 [ 8352.232765] do_filp_open+0x99/0x140 [ 8352.234681] ? getname_flags+0x6e/0x330 [ 8352.236712] ? __check_object_size+0xff/0x256 [ 8352.238752] ? do_raw_spin_unlock+0x75/0x190 [ 8352.240885] ? _raw_spin_unlock+0x12/0x30 [ 8352.242169] do_sys_openat2+0x2b4/0x410 [ 8352.243381] do_sys_open+0x73/0xa0 [ 8352.244345] __x64_sys_openat+0x24/0x30 [ 8352.245557] do_syscall_64+0xc1/0x440 [ 8352.247174] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8352.248686] RIP: 0033:0x7fdc0afbf332 [ 8352.250336] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8352.256313] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8352.259363] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8352.261523] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8352.264440] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8352.266816] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8352.269331] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8356.054801] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 8356.062801] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8356.096907] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8356.108634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8356.124727] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8356.126379] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8356.145625] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8356.149352] RAX: 0000000000007d10 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8356.153507] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8356.157813] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8356.162364] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8356.164600] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8356.168035] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8356.171669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8356.174335] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8356.179250] Call Trace: [ 8356.180345] [ 8356.181381] ? show_regs.cold.9+0x22/0x2f [ 8356.183378] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8356.188215] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8356.193983] ? watchdog+0x40/0x40 [ 8356.201057] ? hrtimer_interrupt+0x128/0x2c0 [ 8356.204175] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8356.209173] ? apic_timer_interrupt+0xf/0x20 [ 8356.211284] [ 8356.217622] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8356.220727] do_raw_spin_lock+0x177/0x190 [ 8356.226505] _raw_spin_lock+0x21/0x30 [ 8356.230387] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8356.233922] ? do_raw_spin_unlock+0x75/0x190 [ 8356.237225] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8356.242207] ? do_wait_intr+0xf0/0xf0 [ 8356.245145] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8356.251526] mdc_close+0x53b/0xd70 [mdc] [ 8356.253937] lmv_close+0x18d/0x490 [lmv] [ 8356.256578] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8356.261887] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8356.268130] ll_md_close+0x274/0x800 [lustre] [ 8356.272513] ll_file_release+0x128/0xc50 [lustre] [ 8356.277861] __fput+0x10d/0x440 [ 8356.279840] ____fput+0x12/0x20 [ 8356.282178] task_work_run+0xc6/0x110 [ 8356.284666] exit_to_usermode_loop+0x1dd/0x1f0 [ 8356.288474] do_syscall_64+0x3d6/0x440 [ 8356.292883] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8356.296265] RIP: 0033:0x7fdc0afbfcd8 [ 8356.297194] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8356.313762] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8356.322192] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8356.324865] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8356.328551] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8356.331867] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8356.335031] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8364.048130] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8364.050337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8364.073539] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8364.079000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8364.085052] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8364.087770] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8364.098212] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8364.101852] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8364.105454] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8364.108785] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8364.112400] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8364.116559] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8364.120475] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8364.124570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8364.127677] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8364.131205] Call Trace: [ 8364.132616] [ 8364.133663] ? show_regs.cold.9+0x22/0x2f [ 8364.135011] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8364.136711] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8364.138217] ? watchdog+0x40/0x40 [ 8364.139877] ? hrtimer_interrupt+0x128/0x2c0 [ 8364.141509] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8364.143734] ? apic_timer_interrupt+0xf/0x20 [ 8364.145950] [ 8364.147189] ? native_safe_halt+0x2e/0x50 [ 8364.149402] kvm_wait+0xcd/0xe0 [ 8364.151091] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8364.154040] ? dump_stack+0xf6/0x10e [ 8364.155650] do_raw_spin_lock+0x177/0x190 [ 8364.158420] _raw_spin_lock+0x21/0x30 [ 8364.159927] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8364.162731] obd_mod_free+0x46/0x140 [ptlrpc] [ 8364.165462] mdc_commit_open+0x91/0xb0 [mdc] [ 8364.167492] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8364.170700] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8364.173368] after_reply+0xe2c/0x1780 [ptlrpc] [ 8364.175170] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8364.177764] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8364.180229] ? do_wait_intr+0xf0/0xf0 [ 8364.181981] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8364.185116] kthread+0x1d1/0x200 [ 8364.186912] ? set_kthread_struct+0x70/0x70 [ 8364.188729] ret_from_fork+0x1f/0x30 [ 8368.051883] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8368.058936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8368.127772] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8368.138798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8368.151570] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8368.158637] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8368.162960] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8368.175053] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8368.184809] RAX: 0000000000005f0f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8368.191347] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8368.195262] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8368.204321] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8368.208804] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8368.213489] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8368.219799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8368.223472] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8368.226887] Call Trace: [ 8368.227688] [ 8368.228431] ? show_regs.cold.9+0x22/0x2f [ 8368.229924] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8368.234426] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8368.236621] ? watchdog+0x40/0x40 [ 8368.237791] ? hrtimer_interrupt+0x128/0x2c0 [ 8368.239710] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8368.241838] ? apic_timer_interrupt+0xf/0x20 [ 8368.246853] [ 8368.247763] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8368.250147] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8368.255814] do_raw_spin_lock+0x177/0x190 [ 8368.258041] _raw_spin_lock+0x21/0x30 [ 8368.259710] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8368.262739] process_one_work+0x2c8/0x700 [ 8368.264707] worker_thread+0x296/0x6e0 [ 8368.266586] ? rescuer_thread+0x570/0x570 [ 8368.268589] kthread+0x1d1/0x200 [ 8368.270503] ? set_kthread_struct+0x70/0x70 [ 8368.277142] ret_from_fork+0x1f/0x30 [ 8369.633484] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6344s! [ 8369.642236] Showing busy workqueues and worker pools: [ 8369.646703] workqueue events: flags=0x0 [ 8369.652399] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8369.662395] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8369.666753] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8369.676976] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8369.697855] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8369.705516] pending: kfree_rcu_monitor [ 8369.707833] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8369.713646] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8369.727479] workqueue events_long: flags=0x0 [ 8369.729205] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8369.740899] pending: lu_objects_flush [obdclass] [ 8369.743868] workqueue events_unbound: flags=0x2 [ 8369.748830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8369.753730] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8369.761701] workqueue events_power_efficient: flags=0x80 [ 8369.767818] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8369.774752] pending: neigh_periodic_work, check_lifetime [ 8369.779078] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8369.782217] pending: do_cache_clean, neigh_periodic_work [ 8369.783903] workqueue mm_percpu_wq: flags=0x8 [ 8369.785849] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8369.789332] pending: lru_add_drain_per_cpu [ 8369.800684] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8369.809158] pending: vmstat_update, lru_add_drain_per_cpu [ 8369.813679] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8369.826819] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8369.840728] workqueue writeback: flags=0x4a [ 8369.845661] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8369.849214] pending: wb_workfn, wb_workfn [ 8369.853729] workqueue xprtiod: flags=0xa [ 8369.856549] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8369.860824] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8369.874799] workqueue ipv6_addrconf: flags=0x40008 [ 8369.884022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8369.891112] pending: addrconf_verify_work [ 8369.892723] workqueue knbd0-recv: flags=0x1a [ 8369.900983] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8369.910822] in-flight: 655:recv_work [ 8369.912818] workqueue ptlrpc_pinger: flags=0x2 [ 8369.919459] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8369.923148] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8369.928842] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6345s workers=4 idle: 36396 38 11188 [ 8369.934880] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8360s workers=2 idle: 658 [ 8379.498106] rcu: INFO: rcu_sched self-detected stall on CPU [ 8379.500751] rcu: 0-...!: (3989187 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 8379.506475] (t=6360155 jiffies g=270761 q=4180) [ 8379.510965] rcu: rcu_sched kthread starved for 6355087 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 8379.516206] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8379.521119] rcu: RCU grace-period kthread stack dump: [ 8379.524171] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8379.528522] Call Trace: [ 8379.529869] __schedule+0x351/0xcb0 [ 8379.531547] schedule+0xc0/0x180 [ 8379.533193] schedule_timeout+0xb4/0x190 [ 8379.535089] ? do_raw_spin_unlock+0x75/0x190 [ 8379.537335] ? __next_timer_interrupt+0x160/0x160 [ 8379.539661] ? prepare_to_swait_event+0xf3/0x140 [ 8379.543013] rcu_gp_kthread+0x8dd/0x15b0 [ 8379.545137] ? force_qs_rnp+0x2c0/0x2c0 [ 8379.547004] kthread+0x1d1/0x200 [ 8379.549393] ? set_kthread_struct+0x70/0x70 [ 8379.551354] ret_from_fork+0x1f/0x30 [ 8379.554261] rcu: Stack dump where RCU GP kthread last ran: [ 8379.556727] Sending NMI from CPU 0 to CPUs 2: [ 8379.561998] NMI backtrace for cpu 2 [ 8379.562006] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8379.562008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8379.562010] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8379.562012] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8379.562014] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 [ 8379.562017] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8379.562019] RDX: 00000000000069d6 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8379.562021] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8379.562022] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8379.562024] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8379.562025] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8379.562027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8379.562028] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8379.562029] Call Trace: [ 8379.562030] [ 8379.562031] ? show_regs.cold.9+0x22/0x2f [ 8379.562033] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8379.562034] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8379.562035] ? nmi_handle+0xbd/0x210 [ 8379.562036] ? default_do_nmi+0x4b/0x1b0 [ 8379.562038] ? do_nmi+0x280/0x380 [ 8379.562039] ? end_repeat_nmi+0x16/0x1f [ 8379.562040] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8379.562041] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8379.562043] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8379.562044] [ 8379.562046] do_raw_spin_lock+0x177/0x190 [ 8379.562047] _raw_spin_lock+0x21/0x30 [ 8379.562048] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8379.562049] ? do_raw_spin_unlock+0x75/0x190 [ 8379.562050] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8379.562051] ? do_wait_intr+0xf0/0xf0 [ 8379.562052] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8379.562053] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8379.562054] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8379.562056] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8379.562057] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8379.562058] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8379.562059] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8379.562060] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8379.562061] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8379.562062] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8379.562063] lmv_intent_open+0x338/0x1150 [lmv] [ 8379.562064] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8379.562066] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8379.562067] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8379.562068] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8379.562069] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8379.562070] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8379.562071] ? do_raw_spin_unlock+0x75/0x190 [ 8379.562072] ? _raw_spin_unlock+0x12/0x30 [ 8379.562073] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8379.562074] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8379.562075] ll_atomic_open+0x36a/0xd70 [lustre] [ 8379.562076] lookup_open+0x356/0xb40 [ 8379.562077] do_last+0x313/0xfc0 [ 8379.562078] ? nd_jump_root+0xe5/0x160 [ 8379.562079] ? path_init+0x437/0x520 [ 8379.562080] path_openat+0xf7/0x500 [ 8379.562081] ? do_raw_spin_unlock+0x75/0x190 [ 8379.562081] do_filp_open+0x99/0x140 [ 8379.562082] ? getname_flags+0x6e/0x330 [ 8379.562083] ? __check_object_size+0xff/0x256 [ 8379.562085] ? do_raw_spin_unlock+0x75/0x190 [ 8379.562085] ? _raw_spin_unlock+0x12/0x30 [ 8379.562086] do_sys_openat2+0x2b4/0x410 [ 8379.562088] do_sys_open+0x73/0xa0 [ 8379.562089] __x64_sys_openat+0x24/0x30 [ 8379.562090] do_syscall_64+0xc1/0x440 [ 8379.562091] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8379.562092] RIP: 0033:0x7fdc0afbf332 [ 8379.562094] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8379.562096] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8379.562099] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8379.562100] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8379.562101] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8379.562102] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8379.562103] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8379.562479] NMI backtrace for cpu 0 [ 8379.839696] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8379.849017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8379.855615] Call Trace: [ 8379.857037] [ 8379.858206] ? dump_stack+0xbb/0x10e [ 8379.859870] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8379.863107] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8379.868514] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8379.870575] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8379.874062] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8379.875745] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8379.877133] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8379.878901] ? update_process_times+0x6d/0xc0 [ 8379.880999] ? tick_sched_handle.isra.18+0x27/0x90 [ 8379.882733] ? tick_sched_timer+0x53/0xb0 [ 8379.886160] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8379.889523] ? tick_sched_handle.isra.18+0x90/0x90 [ 8379.892136] ? hrtimer_interrupt+0x128/0x2c0 [ 8379.898281] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8379.902682] ? apic_timer_interrupt+0xf/0x20 [ 8379.909159] [ 8379.911661] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8379.914527] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8379.916306] ? dump_stack+0xf6/0x10e [ 8379.917529] ? do_raw_spin_lock+0x177/0x190 [ 8379.919668] ? _raw_spin_lock+0x21/0x30 [ 8379.922284] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8379.925543] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8379.927761] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8379.929685] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8379.932072] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8379.934996] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8379.936937] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8379.939372] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8379.940900] ? do_wait_intr+0xf0/0xf0 [ 8379.942195] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8379.944051] ? kthread+0x1d1/0x200 [ 8379.945223] ? set_kthread_struct+0x70/0x70 [ 8379.947091] ? ret_from_fork+0x1f/0x30 [ 8384.054810] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 8384.064522] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8384.103457] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8384.108398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8384.120503] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8384.128158] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8384.146519] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8384.157141] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8384.160055] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8384.166648] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8384.172520] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8384.177153] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8384.182559] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8384.186061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8384.193417] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8384.199478] Call Trace: [ 8384.201321] [ 8384.204764] ? show_regs.cold.9+0x22/0x2f [ 8384.207204] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8384.210265] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8384.212159] ? watchdog+0x40/0x40 [ 8384.215394] ? hrtimer_interrupt+0x128/0x2c0 [ 8384.221593] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8384.224541] ? apic_timer_interrupt+0xf/0x20 [ 8384.226519] [ 8384.227222] ? native_safe_halt+0x2e/0x50 [ 8384.228834] kvm_wait+0xcd/0xe0 [ 8384.230590] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8384.233040] do_raw_spin_lock+0x177/0x190 [ 8384.238383] _raw_spin_lock+0x21/0x30 [ 8384.244063] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8384.249809] ? do_raw_spin_unlock+0x75/0x190 [ 8384.257572] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8384.262083] ? do_wait_intr+0xf0/0xf0 [ 8384.263775] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8384.268495] mdc_close+0x53b/0xd70 [mdc] [ 8384.270549] lmv_close+0x18d/0x490 [lmv] [ 8384.273790] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8384.277311] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8384.279420] ll_md_close+0x274/0x800 [lustre] [ 8384.284634] ll_file_release+0x128/0xc50 [lustre] [ 8384.293416] __fput+0x10d/0x440 [ 8384.294855] ____fput+0x12/0x20 [ 8384.298231] task_work_run+0xc6/0x110 [ 8384.302804] exit_to_usermode_loop+0x1dd/0x1f0 [ 8384.306168] do_syscall_64+0x3d6/0x440 [ 8384.312673] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8384.317694] RIP: 0033:0x7fdc0afbfcd8 [ 8384.320492] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8384.333689] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8384.339150] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8384.342402] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8384.346472] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8384.353619] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8384.359499] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8396.051033] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8396.057194] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8396.082082] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8396.089439] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8396.093255] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8396.096394] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8396.097862] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8396.112849] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8396.115582] RAX: 0000000000007472 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8396.117942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8396.121371] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8396.124020] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8396.126945] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8396.130996] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8396.136668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8396.140708] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8396.148353] Call Trace: [ 8396.149158] [ 8396.150071] ? show_regs.cold.9+0x22/0x2f [ 8396.152259] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8396.154189] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8396.155906] ? watchdog+0x40/0x40 [ 8396.157870] ? hrtimer_interrupt+0x128/0x2c0 [ 8396.159273] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8396.163405] ? apic_timer_interrupt+0xf/0x20 [ 8396.164873] [ 8396.165581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8396.170824] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8396.175220] do_raw_spin_lock+0x177/0x190 [ 8396.176421] _raw_spin_lock+0x21/0x30 [ 8396.177410] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8396.181189] process_one_work+0x2c8/0x700 [ 8396.183257] worker_thread+0x296/0x6e0 [ 8396.185852] ? rescuer_thread+0x570/0x570 [ 8396.193784] kthread+0x1d1/0x200 [ 8396.195659] ? set_kthread_struct+0x70/0x70 [ 8396.197937] ret_from_fork+0x1f/0x30 [ 8400.351563] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6375s! [ 8400.356879] Showing busy workqueues and worker pools: [ 8400.358943] workqueue events: flags=0x0 [ 8400.360886] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8400.364032] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8400.367350] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8400.371215] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8400.376028] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8400.379446] pending: kfree_rcu_monitor [ 8400.381501] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8400.384643] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8400.391283] workqueue events_long: flags=0x0 [ 8400.393966] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8400.397950] pending: lu_objects_flush [obdclass] [ 8400.400027] workqueue events_unbound: flags=0x2 [ 8400.401862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8400.404888] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8400.409539] workqueue events_power_efficient: flags=0x80 [ 8400.411671] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8400.414566] pending: neigh_periodic_work, check_lifetime [ 8400.416952] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8400.420251] pending: do_cache_clean, neigh_periodic_work [ 8400.423607] workqueue mm_percpu_wq: flags=0x8 [ 8400.425524] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8400.428283] pending: lru_add_drain_per_cpu [ 8400.430764] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8400.433700] pending: vmstat_update, lru_add_drain_per_cpu [ 8400.436403] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8400.439861] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8400.442859] workqueue writeback: flags=0x4a [ 8400.444895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8400.447899] pending: wb_workfn, wb_workfn [ 8400.450152] workqueue xprtiod: flags=0xa [ 8400.452037] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8400.454739] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8400.458555] workqueue ipv6_addrconf: flags=0x40008 [ 8400.460605] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8400.463662] pending: addrconf_verify_work [ 8400.465839] workqueue knbd0-recv: flags=0x1a [ 8400.467909] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8400.471365] in-flight: 655:recv_work [ 8400.473430] workqueue ptlrpc_pinger: flags=0x2 [ 8400.475141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8400.477760] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8400.481446] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6375s workers=4 idle: 36396 38 11188 [ 8400.485292] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8391s workers=2 idle: 658 [ 8404.052008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8404.055635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8404.079895] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8404.086610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8404.091667] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8404.093335] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8404.102985] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8404.106926] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8404.110969] RDX: 0000000000000212 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8404.115015] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8404.119875] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8404.123063] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8404.128153] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8404.133097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8404.136868] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8404.140917] Call Trace: [ 8404.142318] [ 8404.143162] ? show_regs.cold.9+0x22/0x2f [ 8404.145142] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8404.146948] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8404.149022] ? watchdog+0x40/0x40 [ 8404.152408] ? hrtimer_interrupt+0x128/0x2c0 [ 8404.155686] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8404.157694] ? apic_timer_interrupt+0xf/0x20 [ 8404.159837] [ 8404.161110] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8404.163570] do_raw_spin_lock+0x177/0x190 [ 8404.165556] _raw_spin_lock+0x21/0x30 [ 8404.167091] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8404.169893] ? do_raw_spin_unlock+0x75/0x190 [ 8404.171762] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8404.174055] ? do_wait_intr+0xf0/0xf0 [ 8404.175048] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8404.178091] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8404.183486] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8404.190479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8404.193095] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8404.196124] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8404.198758] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8404.200204] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8404.202947] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8404.206996] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8404.209482] lmv_intent_open+0x338/0x1150 [lmv] [ 8404.212676] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8404.215191] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8404.218540] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8404.221292] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8404.224011] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8404.226818] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8404.228958] ? do_raw_spin_unlock+0x75/0x190 [ 8404.231658] ? _raw_spin_unlock+0x12/0x30 [ 8404.233489] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8404.239142] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8404.243708] ll_atomic_open+0x36a/0xd70 [lustre] [ 8404.245861] lookup_open+0x356/0xb40 [ 8404.247730] do_last+0x313/0xfc0 [ 8404.249115] ? nd_jump_root+0xe5/0x160 [ 8404.252017] ? path_init+0x437/0x520 [ 8404.253192] path_openat+0xf7/0x500 [ 8404.254290] ? do_raw_spin_unlock+0x75/0x190 [ 8404.256501] do_filp_open+0x99/0x140 [ 8404.259452] ? getname_flags+0x6e/0x330 [ 8404.261513] ? __check_object_size+0xff/0x256 [ 8404.264828] ? do_raw_spin_unlock+0x75/0x190 [ 8404.266339] ? _raw_spin_unlock+0x12/0x30 [ 8404.268902] do_sys_openat2+0x2b4/0x410 [ 8404.270910] do_sys_open+0x73/0xa0 [ 8404.271828] __x64_sys_openat+0x24/0x30 [ 8404.273955] do_syscall_64+0xc1/0x440 [ 8404.275940] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8404.278261] RIP: 0033:0x7fdc0afbf332 [ 8404.280719] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8404.292632] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8404.296049] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8404.299738] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8404.304307] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8404.308605] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8404.314580] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8412.056270] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 8412.068606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8412.108753] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8412.115716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8412.125770] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8412.132789] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8412.148533] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8412.154815] RAX: 0000000000007e05 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8412.159653] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8412.167696] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8412.175257] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8412.181913] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8412.186038] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8412.197015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8412.200223] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8412.205316] Call Trace: [ 8412.206117] [ 8412.214763] ? show_regs.cold.9+0x22/0x2f [ 8412.216830] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8412.228231] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8412.233840] ? watchdog+0x40/0x40 [ 8412.237759] ? hrtimer_interrupt+0x128/0x2c0 [ 8412.245082] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8412.249974] ? apic_timer_interrupt+0xf/0x20 [ 8412.252648] [ 8412.253770] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8412.257003] do_raw_spin_lock+0x177/0x190 [ 8412.259546] _raw_spin_lock+0x21/0x30 [ 8412.261784] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8412.267691] ? do_raw_spin_unlock+0x75/0x190 [ 8412.269846] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8412.271883] ? do_wait_intr+0xf0/0xf0 [ 8412.274649] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8412.277716] mdc_close+0x53b/0xd70 [mdc] [ 8412.281231] lmv_close+0x18d/0x490 [lmv] [ 8412.283789] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8412.286679] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8412.289788] ll_md_close+0x274/0x800 [lustre] [ 8412.291826] ll_file_release+0x128/0xc50 [lustre] [ 8412.293522] __fput+0x10d/0x440 [ 8412.294730] ____fput+0x12/0x20 [ 8412.295893] task_work_run+0xc6/0x110 [ 8412.298515] exit_to_usermode_loop+0x1dd/0x1f0 [ 8412.301495] do_syscall_64+0x3d6/0x440 [ 8412.303380] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8412.305638] RIP: 0033:0x7fdc0afbfcd8 [ 8412.307143] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8412.317681] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8412.323234] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8412.328656] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8412.336663] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8412.343065] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8412.349660] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8424.051118] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8424.054835] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8424.092921] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8424.096778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8424.102272] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8424.105607] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8424.111352] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8424.129961] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8424.134491] RAX: 0000000000000708 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8424.137684] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8424.140695] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8424.143861] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8424.148302] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8424.155815] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8424.161516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8424.163904] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8424.166297] Call Trace: [ 8424.167312] [ 8424.168428] ? show_regs.cold.9+0x22/0x2f [ 8424.170514] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8424.172738] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8424.174692] ? watchdog+0x40/0x40 [ 8424.176004] ? hrtimer_interrupt+0x128/0x2c0 [ 8424.177569] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8424.179048] ? apic_timer_interrupt+0xf/0x20 [ 8424.180315] [ 8424.180832] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8424.182498] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8424.183849] do_raw_spin_lock+0x177/0x190 [ 8424.184826] _raw_spin_lock+0x21/0x30 [ 8424.186071] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8424.187644] process_one_work+0x2c8/0x700 [ 8424.188612] worker_thread+0x296/0x6e0 [ 8424.189668] ? rescuer_thread+0x570/0x570 [ 8424.190776] kthread+0x1d1/0x200 [ 8424.191622] ? set_kthread_struct+0x70/0x70 [ 8424.192854] ret_from_fork+0x1f/0x30 [ 8428.048055] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 8428.052861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8428.083264] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8428.090321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8428.093968] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8428.097299] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8428.109398] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8428.123946] RAX: 0000000000006d89 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8428.131332] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8428.139128] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8428.152798] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8428.165695] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8428.177155] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8428.182723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8428.189369] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8428.198759] Call Trace: [ 8428.199917] [ 8428.201906] ? show_regs.cold.9+0x22/0x2f [ 8428.205837] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8428.208835] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8428.212388] ? watchdog+0x40/0x40 [ 8428.218759] ? hrtimer_interrupt+0x128/0x2c0 [ 8428.223565] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8428.229382] ? apic_timer_interrupt+0xf/0x20 [ 8428.231451] [ 8428.236754] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8428.238225] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8428.242082] ? dump_stack+0xf6/0x10e [ 8428.245190] do_raw_spin_lock+0x177/0x190 [ 8428.248780] _raw_spin_lock+0x21/0x30 [ 8428.251725] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8428.255464] obd_mod_free+0x46/0x140 [ptlrpc] [ 8428.260174] mdc_commit_open+0x91/0xb0 [mdc] [ 8428.263013] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8428.268476] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8428.271272] after_reply+0xe2c/0x1780 [ptlrpc] [ 8428.273841] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8428.276359] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8428.278421] ? do_wait_intr+0xf0/0xf0 [ 8428.280524] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8428.283187] kthread+0x1d1/0x200 [ 8428.285348] ? set_kthread_struct+0x70/0x70 [ 8428.287734] ret_from_fork+0x1f/0x30 [ 8430.559081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6405s! [ 8430.562664] Showing busy workqueues and worker pools: [ 8430.564625] workqueue events: flags=0x0 [ 8430.566643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8430.571049] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8430.577754] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8430.585019] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8430.591626] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8430.595396] pending: kfree_rcu_monitor [ 8430.597130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8430.600403] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8430.608783] workqueue events_long: flags=0x0 [ 8430.610812] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8430.619458] pending: lu_objects_flush [obdclass] [ 8430.623269] workqueue events_unbound: flags=0x2 [ 8430.625968] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8430.629849] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8430.634987] workqueue events_power_efficient: flags=0x80 [ 8430.638818] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8430.642339] pending: neigh_periodic_work, check_lifetime [ 8430.645844] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8430.649481] pending: do_cache_clean, neigh_periodic_work [ 8430.655018] workqueue mm_percpu_wq: flags=0x8 [ 8430.656513] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8430.659727] pending: lru_add_drain_per_cpu [ 8430.664484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8430.668477] pending: vmstat_update, lru_add_drain_per_cpu [ 8430.671373] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8430.678973] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8430.685827] workqueue writeback: flags=0x4a [ 8430.689153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8430.692321] pending: wb_workfn, wb_workfn [ 8430.694569] workqueue xprtiod: flags=0xa [ 8430.697931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8430.706795] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8430.716040] workqueue ipv6_addrconf: flags=0x40008 [ 8430.720894] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8430.725809] pending: addrconf_verify_work [ 8430.732166] workqueue knbd0-recv: flags=0x1a [ 8430.737808] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8430.741510] in-flight: 655:recv_work [ 8430.742856] workqueue ptlrpc_pinger: flags=0x2 [ 8430.747381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8430.752038] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8430.759550] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6405s workers=4 idle: 36396 38 11188 [ 8430.767936] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8421s workers=2 idle: 658 [ 8432.052363] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8432.057228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8432.082100] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8432.087070] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8432.092271] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8432.095650] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8432.106592] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8432.110838] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8432.114800] RDX: 00000000000011e6 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8432.118398] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8432.122445] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8432.126484] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8432.128587] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8432.132577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8432.136259] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8432.141522] Call Trace: [ 8432.142962] [ 8432.143902] ? show_regs.cold.9+0x22/0x2f [ 8432.146485] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8432.148936] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8432.151121] ? watchdog+0x40/0x40 [ 8432.153004] ? hrtimer_interrupt+0x128/0x2c0 [ 8432.155245] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8432.157589] ? apic_timer_interrupt+0xf/0x20 [ 8432.159735] [ 8432.160772] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8432.163145] do_raw_spin_lock+0x177/0x190 [ 8432.165109] _raw_spin_lock+0x21/0x30 [ 8432.167221] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8432.169905] ? do_raw_spin_unlock+0x75/0x190 [ 8432.172535] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8432.176324] ? do_wait_intr+0xf0/0xf0 [ 8432.178252] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8432.181995] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8432.186486] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8432.189895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8432.194186] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8432.197225] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8432.199623] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8432.201943] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8432.204640] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8432.207897] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8432.210962] lmv_intent_open+0x338/0x1150 [lmv] [ 8432.213251] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8432.216178] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8432.218899] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8432.221184] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8432.223750] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8432.226900] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8432.229329] ? do_raw_spin_unlock+0x75/0x190 [ 8432.231608] ? _raw_spin_unlock+0x12/0x30 [ 8432.233623] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8432.236149] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8432.239151] ll_atomic_open+0x36a/0xd70 [lustre] [ 8432.242044] lookup_open+0x356/0xb40 [ 8432.243778] do_last+0x313/0xfc0 [ 8432.245461] ? nd_jump_root+0xe5/0x160 [ 8432.247156] ? path_init+0x437/0x520 [ 8432.249120] path_openat+0xf7/0x500 [ 8432.250889] ? do_raw_spin_unlock+0x75/0x190 [ 8432.253071] do_filp_open+0x99/0x140 [ 8432.254781] ? getname_flags+0x6e/0x330 [ 8432.256368] ? __check_object_size+0xff/0x256 [ 8432.257937] ? do_raw_spin_unlock+0x75/0x190 [ 8432.260082] ? _raw_spin_unlock+0x12/0x30 [ 8432.262127] do_sys_openat2+0x2b4/0x410 [ 8432.264098] do_sys_open+0x73/0xa0 [ 8432.265730] __x64_sys_openat+0x24/0x30 [ 8432.267589] do_syscall_64+0xc1/0x440 [ 8432.269476] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8432.272063] RIP: 0033:0x7fdc0afbf332 [ 8432.273897] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8432.283142] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8432.287012] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8432.291106] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8432.295309] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8432.298781] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8432.302148] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8440.054052] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8440.056498] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8440.099108] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8440.106180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8440.112665] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8440.115647] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8440.128662] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8440.136122] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8440.142698] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8440.148987] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8440.155435] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8440.159150] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8440.162045] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8440.168672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8440.174502] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8440.179538] Call Trace: [ 8440.180204] [ 8440.180775] ? show_regs.cold.9+0x22/0x2f [ 8440.189508] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8440.191397] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8440.193108] ? watchdog+0x40/0x40 [ 8440.194104] ? hrtimer_interrupt+0x128/0x2c0 [ 8440.197949] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8440.199510] ? apic_timer_interrupt+0xf/0x20 [ 8440.200801] [ 8440.204638] ? native_safe_halt+0x2e/0x50 [ 8440.206202] kvm_wait+0xcd/0xe0 [ 8440.208054] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8440.211655] do_raw_spin_lock+0x177/0x190 [ 8440.213151] _raw_spin_lock+0x21/0x30 [ 8440.214799] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8440.218235] ? do_raw_spin_unlock+0x75/0x190 [ 8440.222489] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8440.225019] ? do_wait_intr+0xf0/0xf0 [ 8440.228049] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8440.236579] mdc_close+0x53b/0xd70 [mdc] [ 8440.238542] lmv_close+0x18d/0x490 [lmv] [ 8440.242011] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8440.244535] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8440.246519] ll_md_close+0x274/0x800 [lustre] [ 8440.249952] ll_file_release+0x128/0xc50 [lustre] [ 8440.251893] __fput+0x10d/0x440 [ 8440.253184] ____fput+0x12/0x20 [ 8440.254522] task_work_run+0xc6/0x110 [ 8440.255709] exit_to_usermode_loop+0x1dd/0x1f0 [ 8440.257899] do_syscall_64+0x3d6/0x440 [ 8440.259056] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8440.260958] RIP: 0033:0x7fdc0afbfcd8 [ 8440.262150] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8440.268428] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8440.270933] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8440.273237] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8440.275878] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8440.277900] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8440.280464] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8452.053447] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8452.065694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8452.124787] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8452.133122] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8452.138014] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8452.141342] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8452.144266] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8452.165117] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8452.170652] RAX: 0000000000006615 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8452.177186] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8452.182096] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8452.187027] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8452.191337] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8452.194958] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8452.201384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8452.209530] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8452.212922] Call Trace: [ 8452.217305] [ 8452.218255] ? show_regs.cold.9+0x22/0x2f [ 8452.226707] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8452.227879] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8452.231030] ? watchdog+0x40/0x40 [ 8452.232687] ? hrtimer_interrupt+0x128/0x2c0 [ 8452.237869] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8452.241113] ? apic_timer_interrupt+0xf/0x20 [ 8452.243251] [ 8452.244272] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8452.246806] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8452.249141] do_raw_spin_lock+0x177/0x190 [ 8452.251255] _raw_spin_lock+0x21/0x30 [ 8452.252998] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8452.260558] process_one_work+0x2c8/0x700 [ 8452.263379] worker_thread+0x296/0x6e0 [ 8452.266775] ? rescuer_thread+0x570/0x570 [ 8452.271410] kthread+0x1d1/0x200 [ 8452.272493] ? set_kthread_struct+0x70/0x70 [ 8452.281866] ret_from_fork+0x1f/0x30 [ 8456.048259] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 8456.059646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8456.120073] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8456.125733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8456.131038] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8456.134782] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8456.146145] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8456.153059] RAX: 0000000000007dae RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8456.161991] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8456.174351] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8456.181909] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8456.186808] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8456.190555] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8456.194970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8456.198030] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8456.209662] Call Trace: [ 8456.210782] [ 8456.211912] ? show_regs.cold.9+0x22/0x2f [ 8456.213953] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8456.216221] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8456.218120] ? watchdog+0x40/0x40 [ 8456.219915] ? hrtimer_interrupt+0x128/0x2c0 [ 8456.221786] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8456.223645] ? apic_timer_interrupt+0xf/0x20 [ 8456.225582] [ 8456.227776] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8456.230580] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8456.238527] ? dump_stack+0xf6/0x10e [ 8456.241172] do_raw_spin_lock+0x177/0x190 [ 8456.243488] _raw_spin_lock+0x21/0x30 [ 8456.245248] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8456.248598] obd_mod_free+0x46/0x140 [ptlrpc] [ 8456.256296] mdc_commit_open+0x91/0xb0 [mdc] [ 8456.258958] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8456.261985] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8456.266503] after_reply+0xe2c/0x1780 [ptlrpc] [ 8456.270074] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8456.273312] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8456.278219] ? do_wait_intr+0xf0/0xf0 [ 8456.280448] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8456.284018] kthread+0x1d1/0x200 [ 8456.288135] ? set_kthread_struct+0x70/0x70 [ 8456.292144] ret_from_fork+0x1f/0x30 [ 8460.054500] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 8460.069054] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8460.119623] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8460.133057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8460.142181] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8460.154843] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8460.178882] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8460.190542] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8460.197270] RDX: 000000000000705b RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8460.201546] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8460.204901] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8460.208811] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8460.212871] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8460.220500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8460.225854] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8460.235763] Call Trace: [ 8460.236614] [ 8460.245916] ? show_regs.cold.9+0x22/0x2f [ 8460.256552] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8460.265563] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8460.271447] ? watchdog+0x40/0x40 [ 8460.275765] ? hrtimer_interrupt+0x128/0x2c0 [ 8460.290459] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8460.295260] ? apic_timer_interrupt+0xf/0x20 [ 8460.303002] [ 8460.312480] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8460.322879] do_raw_spin_lock+0x177/0x190 [ 8460.327776] _raw_spin_lock+0x21/0x30 [ 8460.330944] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8460.336494] ? do_raw_spin_unlock+0x75/0x190 [ 8460.339181] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8460.341845] ? do_wait_intr+0xf0/0xf0 [ 8460.345506] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8460.348769] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8460.353194] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8460.358572] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8460.361846] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8460.366573] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8460.370499] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8460.372765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8460.375865] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8460.378734] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8460.380863] lmv_intent_open+0x338/0x1150 [lmv] [ 8460.382777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8460.388495] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8460.390815] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8460.392605] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8460.394762] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8460.397013] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8460.399514] ? do_raw_spin_unlock+0x75/0x190 [ 8460.401555] ? _raw_spin_unlock+0x12/0x30 [ 8460.403217] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8460.405356] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8460.406907] ll_atomic_open+0x36a/0xd70 [lustre] [ 8460.408666] lookup_open+0x356/0xb40 [ 8460.410479] do_last+0x313/0xfc0 [ 8460.411871] ? nd_jump_root+0xe5/0x160 [ 8460.413641] ? path_init+0x437/0x520 [ 8460.415242] path_openat+0xf7/0x500 [ 8460.416777] ? do_raw_spin_unlock+0x75/0x190 [ 8460.418574] do_filp_open+0x99/0x140 [ 8460.420326] ? getname_flags+0x6e/0x330 [ 8460.421927] ? __check_object_size+0xff/0x256 [ 8460.424549] ? do_raw_spin_unlock+0x75/0x190 [ 8460.426590] ? _raw_spin_unlock+0x12/0x30 [ 8460.428413] do_sys_openat2+0x2b4/0x410 [ 8460.430221] do_sys_open+0x73/0xa0 [ 8460.431629] __x64_sys_openat+0x24/0x30 [ 8460.432801] do_syscall_64+0xc1/0x440 [ 8460.433877] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8460.435760] RIP: 0033:0x7fdc0afbf332 [ 8460.437309] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8460.444653] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8460.447842] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8460.450848] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8460.453892] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8460.457179] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8460.459845] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8461.279043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6436s! [ 8461.283957] Showing busy workqueues and worker pools: [ 8461.287519] workqueue events: flags=0x0 [ 8461.291546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8461.295832] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8461.299815] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8461.303331] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8461.308202] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8461.310783] pending: kfree_rcu_monitor [ 8461.312038] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8461.314234] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8461.318664] workqueue events_long: flags=0x0 [ 8461.319900] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8461.321862] pending: lu_objects_flush [obdclass] [ 8461.323510] workqueue events_unbound: flags=0x2 [ 8461.324868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8461.326678] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8461.330022] workqueue events_power_efficient: flags=0x80 [ 8461.331828] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8461.333935] pending: neigh_periodic_work, check_lifetime [ 8461.335721] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8461.337834] pending: do_cache_clean, neigh_periodic_work [ 8461.339586] workqueue mm_percpu_wq: flags=0x8 [ 8461.340937] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8461.343180] pending: lru_add_drain_per_cpu [ 8461.344655] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8461.346968] pending: vmstat_update, lru_add_drain_per_cpu [ 8461.349710] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8461.352841] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8461.355916] workqueue writeback: flags=0x4a [ 8461.357806] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8461.360770] pending: wb_workfn, wb_workfn [ 8461.362976] workqueue xprtiod: flags=0xa [ 8461.364656] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8461.367531] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8461.371068] workqueue ipv6_addrconf: flags=0x40008 [ 8461.373540] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8461.376905] pending: addrconf_verify_work [ 8461.379088] workqueue knbd0-recv: flags=0x1a [ 8461.381198] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8461.384311] in-flight: 655:recv_work [ 8461.386262] workqueue ptlrpc_pinger: flags=0x2 [ 8461.388504] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8461.391518] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8461.394791] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6436s workers=4 idle: 36396 38 11188 [ 8461.398752] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8451s workers=2 idle: 658 [ 8468.054261] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8468.060657] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8468.087619] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8468.093585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8468.098083] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8468.100660] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8468.108397] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8468.111552] RAX: 0000000000005f8d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8468.114769] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8468.117665] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8468.120815] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8468.123650] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8468.126866] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8468.130150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8468.132918] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8468.137226] Call Trace: [ 8468.138885] [ 8468.139885] ? show_regs.cold.9+0x22/0x2f [ 8468.141875] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8468.144188] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8468.146993] ? watchdog+0x40/0x40 [ 8468.149012] ? hrtimer_interrupt+0x128/0x2c0 [ 8468.151113] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8468.153179] ? apic_timer_interrupt+0xf/0x20 [ 8468.155025] [ 8468.155886] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8468.157757] do_raw_spin_lock+0x177/0x190 [ 8468.159164] _raw_spin_lock+0x21/0x30 [ 8468.160809] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8468.162917] ? do_raw_spin_unlock+0x75/0x190 [ 8468.164387] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8468.166576] ? do_wait_intr+0xf0/0xf0 [ 8468.167882] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8468.170156] mdc_close+0x53b/0xd70 [mdc] [ 8468.171379] lmv_close+0x18d/0x490 [lmv] [ 8468.173054] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8468.175323] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8468.178336] ll_md_close+0x274/0x800 [lustre] [ 8468.179943] ll_file_release+0x128/0xc50 [lustre] [ 8468.181575] __fput+0x10d/0x440 [ 8468.182769] ____fput+0x12/0x20 [ 8468.183726] task_work_run+0xc6/0x110 [ 8468.184787] exit_to_usermode_loop+0x1dd/0x1f0 [ 8468.186130] do_syscall_64+0x3d6/0x440 [ 8468.187261] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8468.188901] RIP: 0033:0x7fdc0afbfcd8 [ 8468.189897] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8468.195849] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8468.198048] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8468.199928] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8468.201971] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8468.204031] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8468.206057] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8480.051279] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8480.056460] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8480.092283] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8480.097026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8480.113758] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8480.119255] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8480.129832] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8480.144976] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8480.147693] RAX: 0000000000000560 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8480.150232] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8480.152218] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8480.159908] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8480.163633] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8480.166200] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8480.174301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8480.175929] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8480.181191] Call Trace: [ 8480.182956] [ 8480.183726] ? show_regs.cold.9+0x22/0x2f [ 8480.185919] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8480.187239] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8480.191459] ? watchdog+0x40/0x40 [ 8480.192463] ? hrtimer_interrupt+0x128/0x2c0 [ 8480.193783] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8480.199823] ? apic_timer_interrupt+0xf/0x20 [ 8480.201614] [ 8480.202403] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8480.204429] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8480.206403] do_raw_spin_lock+0x177/0x190 [ 8480.213725] _raw_spin_lock+0x21/0x30 [ 8480.214938] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8480.218822] process_one_work+0x2c8/0x700 [ 8480.220917] worker_thread+0x296/0x6e0 [ 8480.222712] ? rescuer_thread+0x570/0x570 [ 8480.223998] kthread+0x1d1/0x200 [ 8480.225867] ? set_kthread_struct+0x70/0x70 [ 8480.230330] ret_from_fork+0x1f/0x30 [ 8488.052368] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 8488.053443] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 8488.056662] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8488.056746] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8488.056754] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8488.056757] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8488.056772] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8488.056776] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8488.056780] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8488.056782] RDX: 0000000000002760 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8488.056784] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8488.056786] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8488.056789] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8488.056791] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8488.056794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8488.056796] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8488.056804] Call Trace: [ 8488.056810] [ 8488.056815] ? show_regs.cold.9+0x22/0x2f [ 8488.056824] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8488.056830] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8488.056835] ? watchdog+0x40/0x40 [ 8488.056840] ? hrtimer_interrupt+0x128/0x2c0 [ 8488.056844] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8488.056976] ? apic_timer_interrupt+0xf/0x20 [ 8488.056981] [ 8488.056988] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8488.056994] do_raw_spin_lock+0x177/0x190 [ 8488.056998] _raw_spin_lock+0x21/0x30 [ 8488.057005] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8488.057345] ? do_raw_spin_unlock+0x75/0x190 [ 8488.057351] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8488.057627] ? do_wait_intr+0xf0/0xf0 [ 8488.057637] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8488.058060] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8488.058316] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8488.058548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8488.058741] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8488.059173] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8488.059205] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8488.059233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8488.059389] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8488.059731] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8488.059835] lmv_intent_open+0x338/0x1150 [lmv] [ 8488.059885] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8488.060117] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8488.060213] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8488.060243] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8488.060341] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8488.060422] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8488.060572] ? do_raw_spin_unlock+0x75/0x190 [ 8488.060581] ? _raw_spin_unlock+0x12/0x30 [ 8488.060588] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8488.060616] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8488.060636] ll_atomic_open+0x36a/0xd70 [lustre] [ 8488.060801] lookup_open+0x356/0xb40 [ 8488.060817] do_last+0x313/0xfc0 [ 8488.060823] ? nd_jump_root+0xe5/0x160 [ 8488.060828] ? path_init+0x437/0x520 [ 8488.060836] path_openat+0xf7/0x500 [ 8488.060841] ? do_raw_spin_unlock+0x75/0x190 [ 8488.061059] do_filp_open+0x99/0x140 [ 8488.061069] ? getname_flags+0x6e/0x330 [ 8488.061081] ? __check_object_size+0xff/0x256 [ 8488.061088] ? do_raw_spin_unlock+0x75/0x190 [ 8488.061092] ? _raw_spin_unlock+0x12/0x30 [ 8488.061097] do_sys_openat2+0x2b4/0x410 [ 8488.061102] do_sys_open+0x73/0xa0 [ 8488.061107] __x64_sys_openat+0x24/0x30 [ 8488.061111] do_syscall_64+0xc1/0x440 [ 8488.061118] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8488.061124] RIP: 0033:0x7fdc0afbf332 [ 8488.061139] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8488.061143] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8488.061148] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8488.061150] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8488.061152] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8488.061154] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8488.061156] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8488.648504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8488.701056] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8488.714787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8488.723561] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8488.735096] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8488.760464] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8488.769002] RAX: 0000000000005a88 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8488.777066] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8488.782553] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8488.798075] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8488.807493] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8488.809957] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8488.816945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8488.820137] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8488.824205] Call Trace: [ 8488.825752] [ 8488.826930] ? show_regs.cold.9+0x22/0x2f [ 8488.829182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8488.833256] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8488.838612] ? watchdog+0x40/0x40 [ 8488.840386] ? hrtimer_interrupt+0x128/0x2c0 [ 8488.844101] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8488.852636] ? apic_timer_interrupt+0xf/0x20 [ 8488.858337] [ 8488.859746] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8488.865474] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8488.872670] ? dump_stack+0xf6/0x10e [ 8488.879118] do_raw_spin_lock+0x177/0x190 [ 8488.885017] _raw_spin_lock+0x21/0x30 [ 8488.892671] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8488.899732] obd_mod_free+0x46/0x140 [ptlrpc] [ 8488.911379] mdc_commit_open+0x91/0xb0 [mdc] [ 8488.913072] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8488.929337] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8488.938383] after_reply+0xe2c/0x1780 [ptlrpc] [ 8488.945630] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8488.953317] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8488.962958] ? do_wait_intr+0xf0/0xf0 [ 8488.965062] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8488.970291] kthread+0x1d1/0x200 [ 8488.978487] ? set_kthread_struct+0x70/0x70 [ 8488.982819] ret_from_fork+0x1f/0x30 [ 8491.488575] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6466s! [ 8491.493720] Showing busy workqueues and worker pools: [ 8491.506971] workqueue events: flags=0x0 [ 8491.515635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8491.523803] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8491.529561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8491.538408] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8491.546651] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8491.552523] pending: kfree_rcu_monitor [ 8491.554203] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8491.556012] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8491.568191] workqueue events_long: flags=0x0 [ 8491.575335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8491.580756] pending: lu_objects_flush [obdclass] [ 8491.589694] workqueue events_unbound: flags=0x2 [ 8491.597828] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8491.604288] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8491.613340] workqueue events_power_efficient: flags=0x80 [ 8491.620189] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8491.622713] pending: neigh_periodic_work, check_lifetime [ 8491.628370] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8491.633086] pending: do_cache_clean, neigh_periodic_work [ 8491.637071] workqueue mm_percpu_wq: flags=0x8 [ 8491.641172] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8491.646053] pending: lru_add_drain_per_cpu [ 8491.650029] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8491.658944] pending: vmstat_update, lru_add_drain_per_cpu [ 8491.664539] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8491.672088] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8491.678220] workqueue writeback: flags=0x4a [ 8491.680560] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8491.685540] pending: wb_workfn, wb_workfn [ 8491.691060] workqueue xprtiod: flags=0xa [ 8491.697033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8491.709384] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8491.725393] workqueue ipv6_addrconf: flags=0x40008 [ 8491.733294] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8491.743834] pending: addrconf_verify_work [ 8491.752063] workqueue knbd0-recv: flags=0x1a [ 8491.753943] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8491.767687] in-flight: 655:recv_work [ 8491.769138] workqueue ptlrpc_pinger: flags=0x2 [ 8491.770591] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8491.786546] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8491.799348] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6466s workers=4 idle: 36396 38 11188 [ 8491.805357] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8482s workers=2 idle: 658 [ 8496.056486] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8496.064305] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8496.099396] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8496.118033] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8496.123194] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8496.130987] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8496.153785] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8496.164393] RAX: 0000000000007183 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8496.179932] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8496.190393] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8496.195461] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8496.208822] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8496.220056] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8496.232179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8496.234057] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8496.245247] Call Trace: [ 8496.249628] [ 8496.250288] ? show_regs.cold.9+0x22/0x2f [ 8496.256892] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8496.261840] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8496.266821] ? watchdog+0x40/0x40 [ 8496.268944] ? hrtimer_interrupt+0x128/0x2c0 [ 8496.271572] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8496.275916] ? apic_timer_interrupt+0xf/0x20 [ 8496.278557] [ 8496.280026] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8496.288210] do_raw_spin_lock+0x177/0x190 [ 8496.289487] _raw_spin_lock+0x21/0x30 [ 8496.294628] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8496.301069] ? do_raw_spin_unlock+0x75/0x190 [ 8496.303339] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8496.313625] ? do_wait_intr+0xf0/0xf0 [ 8496.318023] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8496.328636] mdc_close+0x53b/0xd70 [mdc] [ 8496.329912] lmv_close+0x18d/0x490 [lmv] [ 8496.335628] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8496.348300] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8496.355043] ll_md_close+0x274/0x800 [lustre] [ 8496.360244] ll_file_release+0x128/0xc50 [lustre] [ 8496.371479] __fput+0x10d/0x440 [ 8496.375675] ____fput+0x12/0x20 [ 8496.377280] task_work_run+0xc6/0x110 [ 8496.381462] exit_to_usermode_loop+0x1dd/0x1f0 [ 8496.383693] do_syscall_64+0x3d6/0x440 [ 8496.385168] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8496.386705] RIP: 0033:0x7fdc0afbfcd8 [ 8496.387819] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8496.410980] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8496.415471] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8496.420427] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8496.425102] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8496.431694] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8496.435506] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8508.052640] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8508.063111] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8508.084091] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8508.088871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8508.092438] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8508.094884] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8508.097009] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8508.103809] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8508.106805] RAX: 00000000000042be RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8508.109572] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8508.112234] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8508.115276] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8508.118351] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8508.121346] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8508.126443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8508.130032] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8508.133372] Call Trace: [ 8508.134146] [ 8508.134637] ? show_regs.cold.9+0x22/0x2f [ 8508.138996] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8508.140673] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8508.146165] ? watchdog+0x40/0x40 [ 8508.148181] ? hrtimer_interrupt+0x128/0x2c0 [ 8508.150285] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8508.153254] ? apic_timer_interrupt+0xf/0x20 [ 8508.156435] [ 8508.161407] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8508.165962] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8508.168785] do_raw_spin_lock+0x177/0x190 [ 8508.171225] _raw_spin_lock+0x21/0x30 [ 8508.173241] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8508.175200] process_one_work+0x2c8/0x700 [ 8508.177174] worker_thread+0x296/0x6e0 [ 8508.178789] ? rescuer_thread+0x570/0x570 [ 8508.180476] kthread+0x1d1/0x200 [ 8508.181705] ? set_kthread_struct+0x70/0x70 [ 8508.183032] ret_from_fork+0x1f/0x30 [ 8516.048029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 8516.051309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8516.052013] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 8516.074637] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8516.074657] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8516.074661] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8516.074680] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8516.074684] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8516.074690] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8516.074693] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8516.074696] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8516.074698] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8516.074701] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8516.074705] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8516.074708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8516.074710] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8516.074718] Call Trace: [ 8516.074725] [ 8516.074731] ? show_regs.cold.9+0x22/0x2f [ 8516.074743] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8516.074749] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8516.074754] ? watchdog+0x40/0x40 [ 8516.074758] ? hrtimer_interrupt+0x128/0x2c0 [ 8516.074762] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8516.074779] ? apic_timer_interrupt+0xf/0x20 [ 8516.074784] [ 8516.074787] ? native_safe_halt+0x2e/0x50 [ 8516.074791] kvm_wait+0xcd/0xe0 [ 8516.074799] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8516.074807] ? dump_stack+0xf6/0x10e [ 8516.074814] do_raw_spin_lock+0x177/0x190 [ 8516.074818] _raw_spin_lock+0x21/0x30 [ 8516.074826] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8516.075469] obd_mod_free+0x46/0x140 [ptlrpc] [ 8516.075683] mdc_commit_open+0x91/0xb0 [mdc] [ 8516.075728] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8516.075939] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8516.076426] after_reply+0xe2c/0x1780 [ptlrpc] [ 8516.076617] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8516.076822] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8516.077275] ? do_wait_intr+0xf0/0xf0 [ 8516.077285] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8516.077462] kthread+0x1d1/0x200 [ 8516.077470] ? set_kthread_struct+0x70/0x70 [ 8516.077475] ret_from_fork+0x1f/0x30 [ 8516.244819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8516.260273] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8516.264680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8516.268002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8516.269819] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8516.275585] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8516.278531] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8516.281773] RDX: 00000000000011db RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8516.284473] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8516.286449] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8516.290042] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8516.296268] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8516.304821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8516.310232] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8516.313916] Call Trace: [ 8516.315100] [ 8516.316118] ? show_regs.cold.9+0x22/0x2f [ 8516.317553] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8516.319081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8516.320702] ? watchdog+0x40/0x40 [ 8516.322083] ? hrtimer_interrupt+0x128/0x2c0 [ 8516.323780] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8516.325621] ? apic_timer_interrupt+0xf/0x20 [ 8516.327063] [ 8516.327973] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8516.329725] do_raw_spin_lock+0x177/0x190 [ 8516.330970] _raw_spin_lock+0x21/0x30 [ 8516.332538] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8516.334775] ? do_raw_spin_unlock+0x75/0x190 [ 8516.336616] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8516.338830] ? do_wait_intr+0xf0/0xf0 [ 8516.341909] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8516.344237] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8516.349841] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8516.352588] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8516.356075] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8516.359176] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8516.362391] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8516.364100] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8516.378522] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8516.384691] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8516.390428] lmv_intent_open+0x338/0x1150 [lmv] [ 8516.396365] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8516.398877] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8516.402692] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8516.406196] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8516.408526] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8516.410919] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8516.413147] ? do_raw_spin_unlock+0x75/0x190 [ 8516.414902] ? _raw_spin_unlock+0x12/0x30 [ 8516.416574] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8516.421751] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8516.423778] ll_atomic_open+0x36a/0xd70 [lustre] [ 8516.426564] lookup_open+0x356/0xb40 [ 8516.428463] do_last+0x313/0xfc0 [ 8516.429930] ? nd_jump_root+0xe5/0x160 [ 8516.431957] ? path_init+0x437/0x520 [ 8516.433796] path_openat+0xf7/0x500 [ 8516.435873] ? do_raw_spin_unlock+0x75/0x190 [ 8516.438063] do_filp_open+0x99/0x140 [ 8516.439797] ? getname_flags+0x6e/0x330 [ 8516.440914] ? __check_object_size+0xff/0x256 [ 8516.441971] ? do_raw_spin_unlock+0x75/0x190 [ 8516.443052] ? _raw_spin_unlock+0x12/0x30 [ 8516.444457] do_sys_openat2+0x2b4/0x410 [ 8516.445731] do_sys_open+0x73/0xa0 [ 8516.447226] __x64_sys_openat+0x24/0x30 [ 8516.448469] do_syscall_64+0xc1/0x440 [ 8516.451669] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8516.457770] RIP: 0033:0x7fdc0afbf332 [ 8516.462468] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8516.483479] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8516.490926] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8516.504324] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8516.512709] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8516.516663] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8516.521960] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8522.207550] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6497s! [ 8522.210759] Showing busy workqueues and worker pools: [ 8522.214973] workqueue events: flags=0x0 [ 8522.217113] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8522.219744] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8522.223876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8522.228865] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8522.236488] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8522.239919] pending: kfree_rcu_monitor [ 8522.241773] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8522.244133] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8522.249332] workqueue events_long: flags=0x0 [ 8522.251153] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8522.253751] pending: lu_objects_flush [obdclass] [ 8522.255858] workqueue events_unbound: flags=0x2 [ 8522.257454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8522.260197] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8522.264342] workqueue events_power_efficient: flags=0x80 [ 8522.266368] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8522.268904] pending: neigh_periodic_work, check_lifetime [ 8522.271013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8522.273502] pending: do_cache_clean, neigh_periodic_work [ 8522.276058] workqueue mm_percpu_wq: flags=0x8 [ 8522.277789] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8522.280161] pending: lru_add_drain_per_cpu [ 8522.281916] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8522.284389] pending: vmstat_update, lru_add_drain_per_cpu [ 8522.286638] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8522.289087] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8522.291389] workqueue writeback: flags=0x4a [ 8522.293014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8522.295376] pending: wb_workfn, wb_workfn [ 8522.296906] workqueue xprtiod: flags=0xa [ 8522.298314] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8522.301011] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8522.304343] workqueue ipv6_addrconf: flags=0x40008 [ 8522.306178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8522.308908] pending: addrconf_verify_work [ 8522.310393] workqueue knbd0-recv: flags=0x1a [ 8522.311989] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8522.314205] in-flight: 655:recv_work [ 8522.316215] workqueue ptlrpc_pinger: flags=0x2 [ 8522.317890] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8522.323153] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8522.328126] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6497s workers=4 idle: 36396 38 11188 [ 8522.334140] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8512s workers=2 idle: 658 [ 8524.054787] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8524.057903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8524.073864] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8524.077838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8524.081162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8524.083195] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8524.089887] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8524.093297] RAX: 0000000000001384 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8524.096545] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8524.100496] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8524.104209] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8524.109294] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8524.112700] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8524.116411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8524.119292] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8524.122504] Call Trace: [ 8524.123497] [ 8524.124204] ? show_regs.cold.9+0x22/0x2f [ 8524.125997] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8524.127799] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8524.129520] ? watchdog+0x40/0x40 [ 8524.130786] ? hrtimer_interrupt+0x128/0x2c0 [ 8524.132563] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8524.135041] ? apic_timer_interrupt+0xf/0x20 [ 8524.137325] [ 8524.138520] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8524.141297] do_raw_spin_lock+0x177/0x190 [ 8524.143264] _raw_spin_lock+0x21/0x30 [ 8524.145154] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8524.148202] ? do_raw_spin_unlock+0x75/0x190 [ 8524.150541] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8524.153239] ? do_wait_intr+0xf0/0xf0 [ 8524.154931] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8524.157534] mdc_close+0x53b/0xd70 [mdc] [ 8524.158905] lmv_close+0x18d/0x490 [lmv] [ 8524.160607] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8524.162914] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8524.164975] ll_md_close+0x274/0x800 [lustre] [ 8524.167109] ll_file_release+0x128/0xc50 [lustre] [ 8524.168918] __fput+0x10d/0x440 [ 8524.170467] ____fput+0x12/0x20 [ 8524.172030] task_work_run+0xc6/0x110 [ 8524.173844] exit_to_usermode_loop+0x1dd/0x1f0 [ 8524.175874] do_syscall_64+0x3d6/0x440 [ 8524.177407] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8524.180008] RIP: 0033:0x7fdc0afbfcd8 [ 8524.181820] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8524.190307] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8524.194039] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8524.197713] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8524.201029] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8524.204274] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8524.207321] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8536.051789] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8536.062748] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8536.112665] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8536.122216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8536.132044] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8536.137723] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8536.142772] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8536.156079] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8536.161610] RAX: 000000000000369c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8536.164083] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8536.173809] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8536.194734] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8536.202656] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8536.208856] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8536.214427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8536.223679] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8536.228633] Call Trace: [ 8536.229965] [ 8536.230791] ? show_regs.cold.9+0x22/0x2f [ 8536.232270] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8536.234604] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8536.236650] ? watchdog+0x40/0x40 [ 8536.238232] ? hrtimer_interrupt+0x128/0x2c0 [ 8536.240349] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8536.242555] ? apic_timer_interrupt+0xf/0x20 [ 8536.244606] [ 8536.245609] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8536.247603] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8536.250198] do_raw_spin_lock+0x177/0x190 [ 8536.251861] _raw_spin_lock+0x21/0x30 [ 8536.253632] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8536.256601] process_one_work+0x2c8/0x700 [ 8536.258551] worker_thread+0x296/0x6e0 [ 8536.260305] ? rescuer_thread+0x570/0x570 [ 8536.262361] kthread+0x1d1/0x200 [ 8536.263756] ? set_kthread_struct+0x70/0x70 [ 8536.265733] ret_from_fork+0x1f/0x30 [ 8544.054941] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 8544.059802] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8544.092063] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8544.101396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8544.109142] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8544.114809] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8544.139916] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8544.148160] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8544.155542] RDX: 00000000000012eb RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8544.164823] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8544.177144] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8544.184735] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8544.196024] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8544.204694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8544.210742] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8544.220597] Call Trace: [ 8544.223763] [ 8544.224408] ? show_regs.cold.9+0x22/0x2f [ 8544.225482] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8544.230289] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8544.237085] ? watchdog+0x40/0x40 [ 8544.238283] ? hrtimer_interrupt+0x128/0x2c0 [ 8544.244205] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8544.246308] ? apic_timer_interrupt+0xf/0x20 [ 8544.250336] [ 8544.253789] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8544.255761] do_raw_spin_lock+0x177/0x190 [ 8544.256936] _raw_spin_lock+0x21/0x30 [ 8544.267851] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8544.269543] ? do_raw_spin_unlock+0x75/0x190 [ 8544.275026] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8544.282215] ? do_wait_intr+0xf0/0xf0 [ 8544.286702] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8544.292983] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8544.299390] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8544.302522] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8544.304251] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8544.318757] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8544.321759] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8544.323294] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8544.325748] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8544.327613] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8544.334047] lmv_intent_open+0x338/0x1150 [lmv] [ 8544.336666] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8544.339625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8544.342038] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8544.353316] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8544.357219] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8544.362592] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8544.365817] ? do_raw_spin_unlock+0x75/0x190 [ 8544.370741] ? _raw_spin_unlock+0x12/0x30 [ 8544.373326] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8544.378298] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8544.379922] ll_atomic_open+0x36a/0xd70 [lustre] [ 8544.384084] lookup_open+0x356/0xb40 [ 8544.385964] do_last+0x313/0xfc0 [ 8544.386945] ? nd_jump_root+0xe5/0x160 [ 8544.389666] ? path_init+0x437/0x520 [ 8544.390827] path_openat+0xf7/0x500 [ 8544.391841] ? do_raw_spin_unlock+0x75/0x190 [ 8544.396410] do_filp_open+0x99/0x140 [ 8544.398099] ? getname_flags+0x6e/0x330 [ 8544.405282] ? __check_object_size+0xff/0x256 [ 8544.407004] ? do_raw_spin_unlock+0x75/0x190 [ 8544.408715] ? _raw_spin_unlock+0x12/0x30 [ 8544.410501] do_sys_openat2+0x2b4/0x410 [ 8544.412024] do_sys_open+0x73/0xa0 [ 8544.413524] __x64_sys_openat+0x24/0x30 [ 8544.414835] do_syscall_64+0xc1/0x440 [ 8544.416700] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8544.429328] RIP: 0033:0x7fdc0afbf332 [ 8544.432731] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8544.444175] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8544.447660] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8544.453417] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8544.460011] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8544.462247] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8544.465290] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8548.048066] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8548.052558] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8548.076588] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8548.082182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8548.086323] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8548.088949] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8548.097666] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8548.101527] RAX: 0000000000005706 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8548.104400] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8548.108289] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8548.112430] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8548.116444] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8548.120781] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8548.123995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8548.126197] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8548.129909] Call Trace: [ 8548.130960] [ 8548.131718] ? show_regs.cold.9+0x22/0x2f [ 8548.133717] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8548.135668] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8548.137980] ? watchdog+0x40/0x40 [ 8548.139511] ? hrtimer_interrupt+0x128/0x2c0 [ 8548.141528] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8548.143797] ? apic_timer_interrupt+0xf/0x20 [ 8548.147750] [ 8548.148742] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8548.151592] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8548.155151] ? dump_stack+0xf6/0x10e [ 8548.158136] do_raw_spin_lock+0x177/0x190 [ 8548.163525] _raw_spin_lock+0x21/0x30 [ 8548.165476] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8548.171362] obd_mod_free+0x46/0x140 [ptlrpc] [ 8548.175215] mdc_commit_open+0x91/0xb0 [mdc] [ 8548.177826] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8548.180845] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8548.184327] after_reply+0xe2c/0x1780 [ptlrpc] [ 8548.187785] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8548.191282] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8548.194302] ? do_wait_intr+0xf0/0xf0 [ 8548.201656] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8548.204935] kthread+0x1d1/0x200 [ 8548.206684] ? set_kthread_struct+0x70/0x70 [ 8548.209126] ret_from_fork+0x1f/0x30 [ 8552.054944] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8552.062780] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8552.090933] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8552.096659] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8552.100754] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8552.103706] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8552.112950] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8552.116123] RAX: 0000000000004991 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8552.119487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8552.122915] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8552.125581] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8552.128573] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8552.132255] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8552.136562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8552.139299] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8552.143019] Call Trace: [ 8552.143973] [ 8552.144721] ? show_regs.cold.9+0x22/0x2f [ 8552.146616] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8552.149016] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8552.150899] ? watchdog+0x40/0x40 [ 8552.152100] ? hrtimer_interrupt+0x128/0x2c0 [ 8552.154033] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8552.156280] ? apic_timer_interrupt+0xf/0x20 [ 8552.158075] [ 8552.159048] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8552.161217] do_raw_spin_lock+0x177/0x190 [ 8552.162920] _raw_spin_lock+0x21/0x30 [ 8552.164356] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8552.166455] ? do_raw_spin_unlock+0x75/0x190 [ 8552.168303] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8552.170964] ? do_wait_intr+0xf0/0xf0 [ 8552.171968] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8552.173799] mdc_close+0x53b/0xd70 [mdc] [ 8552.175833] lmv_close+0x18d/0x490 [lmv] [ 8552.177215] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8552.179228] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8552.181300] ll_md_close+0x274/0x800 [lustre] [ 8552.183090] ll_file_release+0x128/0xc50 [lustre] [ 8552.184901] __fput+0x10d/0x440 [ 8552.185921] ____fput+0x12/0x20 [ 8552.186967] task_work_run+0xc6/0x110 [ 8552.188084] exit_to_usermode_loop+0x1dd/0x1f0 [ 8552.189534] do_syscall_64+0x3d6/0x440 [ 8552.190696] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8552.192219] RIP: 0033:0x7fdc0afbfcd8 [ 8552.193245] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8552.198896] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8552.200909] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8552.202989] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8552.205247] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8552.207180] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8552.209152] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8552.419525] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6527s! [ 8552.425314] Showing busy workqueues and worker pools: [ 8552.432248] workqueue events: flags=0x0 [ 8552.439387] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8552.444059] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8552.450588] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8552.456263] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8552.462079] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8552.466489] pending: kfree_rcu_monitor [ 8552.468309] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8552.472574] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8552.482646] workqueue events_long: flags=0x0 [ 8552.488722] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8552.497330] pending: lu_objects_flush [obdclass] [ 8552.507239] workqueue events_unbound: flags=0x2 [ 8552.515155] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8552.519296] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8552.530150] workqueue events_power_efficient: flags=0x80 [ 8552.535660] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8552.540563] pending: neigh_periodic_work, check_lifetime [ 8552.547961] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8552.555678] pending: do_cache_clean, neigh_periodic_work [ 8552.561700] workqueue mm_percpu_wq: flags=0x8 [ 8552.568274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8552.577077] pending: lru_add_drain_per_cpu [ 8552.579757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8552.583141] pending: vmstat_update, lru_add_drain_per_cpu [ 8552.590875] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8552.595823] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8552.600467] workqueue writeback: flags=0x4a [ 8552.608680] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8552.617084] pending: wb_workfn, wb_workfn [ 8552.618434] workqueue xprtiod: flags=0xa [ 8552.627596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8552.636229] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8552.643615] workqueue ipv6_addrconf: flags=0x40008 [ 8552.647294] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8552.652765] pending: addrconf_verify_work [ 8552.654245] workqueue knbd0-recv: flags=0x1a [ 8552.660611] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8552.664675] in-flight: 655:recv_work [ 8552.677148] workqueue ptlrpc_pinger: flags=0x2 [ 8552.679754] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8552.687685] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8552.689646] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6527s workers=4 idle: 36396 38 11188 [ 8552.709700] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8543s workers=2 idle: 658 [ 8559.501500] rcu: INFO: rcu_sched self-detected stall on CPU [ 8559.503796] rcu: 0-...!: (4111694 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 8559.512827] (t=6540158 jiffies g=270761 q=4214) [ 8559.514833] rcu: rcu_sched kthread starved for 6535090 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 8559.530523] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8559.539161] rcu: RCU grace-period kthread stack dump: [ 8559.544454] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8559.551212] Call Trace: [ 8559.552354] __schedule+0x351/0xcb0 [ 8559.560756] schedule+0xc0/0x180 [ 8559.561991] schedule_timeout+0xb4/0x190 [ 8559.568096] ? do_raw_spin_unlock+0x75/0x190 [ 8559.569998] ? __next_timer_interrupt+0x160/0x160 [ 8559.573782] ? prepare_to_swait_event+0xf3/0x140 [ 8559.576813] rcu_gp_kthread+0x8dd/0x15b0 [ 8559.579553] ? force_qs_rnp+0x2c0/0x2c0 [ 8559.588242] kthread+0x1d1/0x200 [ 8559.589842] ? set_kthread_struct+0x70/0x70 [ 8559.595537] ret_from_fork+0x1f/0x30 [ 8559.597794] rcu: Stack dump where RCU GP kthread last ran: [ 8559.602522] Sending NMI from CPU 0 to CPUs 1: [ 8559.607419] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 8559.609443] NMI backtrace for cpu 0 [ 8559.618103] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8559.629749] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8559.637729] Call Trace: [ 8559.639282] [ 8559.640392] ? dump_stack+0xbb/0x10e [ 8559.641997] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8559.644048] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8559.646292] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8559.648397] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8559.650549] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8559.652725] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8559.654175] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8559.656074] ? update_process_times+0x6d/0xc0 [ 8559.658701] ? tick_sched_handle.isra.18+0x27/0x90 [ 8559.660894] ? tick_sched_timer+0x53/0xb0 [ 8559.662576] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8559.664413] ? tick_sched_handle.isra.18+0x90/0x90 [ 8559.666437] ? hrtimer_interrupt+0x128/0x2c0 [ 8559.668814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8559.670792] ? apic_timer_interrupt+0xf/0x20 [ 8559.672682] [ 8559.673547] ? native_safe_halt+0x2e/0x50 [ 8559.675308] ? kvm_wait+0xcd/0xe0 [ 8559.676709] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8559.682829] ? dump_stack+0xf6/0x10e [ 8559.688274] ? do_raw_spin_lock+0x177/0x190 [ 8559.694513] ? _raw_spin_lock+0x21/0x30 [ 8559.698150] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8559.706728] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8559.712048] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8559.715794] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8559.722884] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8559.726849] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8559.734851] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8559.743437] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8559.746070] ? do_wait_intr+0xf0/0xf0 [ 8559.748035] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8559.751505] ? kthread+0x1d1/0x200 [ 8559.755231] ? set_kthread_struct+0x70/0x70 [ 8559.758280] ? ret_from_fork+0x1f/0x30 [ 8572.052834] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 8572.055305] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8572.131854] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8572.145688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8572.156772] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8572.161411] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8572.176828] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8572.181772] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8572.186856] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8572.194191] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8572.205953] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8572.211466] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8572.219154] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8572.224739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8572.229890] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8572.238280] Call Trace: [ 8572.239319] [ 8572.240380] ? show_regs.cold.9+0x22/0x2f [ 8572.241908] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8572.245094] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8572.246720] ? watchdog+0x40/0x40 [ 8572.248677] ? hrtimer_interrupt+0x128/0x2c0 [ 8572.252931] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8572.254815] ? apic_timer_interrupt+0xf/0x20 [ 8572.256273] [ 8572.257113] ? native_safe_halt+0x2e/0x50 [ 8572.258545] kvm_wait+0xcd/0xe0 [ 8572.259730] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8572.267555] do_raw_spin_lock+0x177/0x190 [ 8572.269581] _raw_spin_lock+0x21/0x30 [ 8572.273623] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8572.276188] ? do_raw_spin_unlock+0x75/0x190 [ 8572.277538] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8572.285496] ? do_wait_intr+0xf0/0xf0 [ 8572.286741] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8572.301482] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8572.306883] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8572.309945] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8572.313105] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8572.317926] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8572.323749] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8572.328376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8572.335330] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8572.341603] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8572.343514] lmv_intent_open+0x338/0x1150 [lmv] [ 8572.346254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8572.349100] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8572.352281] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8572.354652] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8572.357904] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8572.361069] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8572.368766] ? do_raw_spin_unlock+0x75/0x190 [ 8572.371325] ? _raw_spin_unlock+0x12/0x30 [ 8572.376307] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8572.381944] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8572.391097] ll_atomic_open+0x36a/0xd70 [lustre] [ 8572.399658] lookup_open+0x356/0xb40 [ 8572.404688] do_last+0x313/0xfc0 [ 8572.407834] ? nd_jump_root+0xe5/0x160 [ 8572.415521] ? path_init+0x437/0x520 [ 8572.419705] path_openat+0xf7/0x500 [ 8572.422592] ? do_raw_spin_unlock+0x75/0x190 [ 8572.425043] do_filp_open+0x99/0x140 [ 8572.426844] ? getname_flags+0x6e/0x330 [ 8572.430006] ? __check_object_size+0xff/0x256 [ 8572.431921] ? do_raw_spin_unlock+0x75/0x190 [ 8572.445710] ? _raw_spin_unlock+0x12/0x30 [ 8572.451587] do_sys_openat2+0x2b4/0x410 [ 8572.456586] do_sys_open+0x73/0xa0 [ 8572.460291] __x64_sys_openat+0x24/0x30 [ 8572.466185] do_syscall_64+0xc1/0x440 [ 8572.467550] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8572.474415] RIP: 0033:0x7fdc0afbf332 [ 8572.476652] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8572.489199] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8572.496739] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8572.504862] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8572.511633] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8572.515500] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8572.525409] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8580.054106] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8580.058512] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8580.078189] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8580.082564] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8580.086604] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8580.089613] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8580.098217] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8580.101357] RAX: 0000000000006e82 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8580.104596] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8580.108782] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8580.111606] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8580.113881] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8580.116587] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8580.119325] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8580.121612] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8580.124850] Call Trace: [ 8580.125838] [ 8580.126487] ? show_regs.cold.9+0x22/0x2f [ 8580.128374] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8580.130456] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8580.132723] ? watchdog+0x40/0x40 [ 8580.134082] ? hrtimer_interrupt+0x128/0x2c0 [ 8580.135905] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8580.137894] ? apic_timer_interrupt+0xf/0x20 [ 8580.139670] [ 8580.140592] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8580.142862] do_raw_spin_lock+0x177/0x190 [ 8580.144835] _raw_spin_lock+0x21/0x30 [ 8580.146900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8580.149367] ? do_raw_spin_unlock+0x75/0x190 [ 8580.151198] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8580.153778] ? do_wait_intr+0xf0/0xf0 [ 8580.155439] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8580.158107] mdc_close+0x53b/0xd70 [mdc] [ 8580.160228] lmv_close+0x18d/0x490 [lmv] [ 8580.161849] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8580.164009] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8580.165612] ll_md_close+0x274/0x800 [lustre] [ 8580.167786] ll_file_release+0x128/0xc50 [lustre] [ 8580.169483] __fput+0x10d/0x440 [ 8580.170992] ____fput+0x12/0x20 [ 8580.172701] task_work_run+0xc6/0x110 [ 8580.174490] exit_to_usermode_loop+0x1dd/0x1f0 [ 8580.176678] do_syscall_64+0x3d6/0x440 [ 8580.178097] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8580.180506] RIP: 0033:0x7fdc0afbfcd8 [ 8580.182145] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8580.189565] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8580.193076] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8580.196360] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8580.199661] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8580.202890] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8580.206342] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8583.135084] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6558s! [ 8583.138326] Showing busy workqueues and worker pools: [ 8583.140744] workqueue events: flags=0x0 [ 8583.142558] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8583.146086] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8583.149648] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8583.152802] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8583.156925] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8583.159989] pending: kfree_rcu_monitor [ 8583.161846] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8583.164441] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8583.169795] workqueue events_long: flags=0x0 [ 8583.171777] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8583.174164] pending: lu_objects_flush [obdclass] [ 8583.175995] workqueue events_unbound: flags=0x2 [ 8583.178297] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8583.181023] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8583.185432] workqueue events_power_efficient: flags=0x80 [ 8583.187865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8583.190425] pending: neigh_periodic_work, check_lifetime [ 8583.192807] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8583.196116] pending: do_cache_clean, neigh_periodic_work [ 8583.198230] workqueue mm_percpu_wq: flags=0x8 [ 8583.200054] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8583.203258] pending: lru_add_drain_per_cpu [ 8583.205164] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8583.207464] pending: vmstat_update, lru_add_drain_per_cpu [ 8583.212085] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8583.214917] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8583.217742] workqueue writeback: flags=0x4a [ 8583.219712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8583.222276] pending: wb_workfn, wb_workfn [ 8583.223877] workqueue xprtiod: flags=0xa [ 8583.225862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8583.228021] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8583.231122] workqueue ipv6_addrconf: flags=0x40008 [ 8583.232887] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8583.235291] pending: addrconf_verify_work [ 8583.236844] workqueue knbd0-recv: flags=0x1a [ 8583.238441] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8583.240856] in-flight: 655:recv_work [ 8583.242498] workqueue ptlrpc_pinger: flags=0x2 [ 8583.243775] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8583.246050] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8583.248530] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6558s workers=4 idle: 36396 38 11188 [ 8583.250955] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8573s workers=2 idle: 658 [ 8584.051022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8584.054814] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8584.075132] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8584.081644] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8584.085770] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8584.089016] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8584.091868] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8584.101740] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8584.105046] RAX: 00000000000014bd RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8584.108802] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8584.111614] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8584.115880] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8584.120505] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8584.124106] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8584.127745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8584.129894] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8584.132753] Call Trace: [ 8584.133707] [ 8584.134175] ? show_regs.cold.9+0x22/0x2f [ 8584.136745] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8584.138208] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8584.141711] ? watchdog+0x40/0x40 [ 8584.142921] ? hrtimer_interrupt+0x128/0x2c0 [ 8584.144958] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8584.146897] ? apic_timer_interrupt+0xf/0x20 [ 8584.148799] [ 8584.149411] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8584.151459] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8584.153237] do_raw_spin_lock+0x177/0x190 [ 8584.154721] _raw_spin_lock+0x21/0x30 [ 8584.156084] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8584.157996] process_one_work+0x2c8/0x700 [ 8584.159776] worker_thread+0x296/0x6e0 [ 8584.160969] ? rescuer_thread+0x570/0x570 [ 8584.162525] kthread+0x1d1/0x200 [ 8584.163554] ? set_kthread_struct+0x70/0x70 [ 8584.165788] ret_from_fork+0x1f/0x30 [ 8600.056126] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8600.063304] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8600.106739] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8600.113517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8600.121532] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8600.123690] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8600.131791] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8600.136471] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8600.139481] RDX: 000000000000372e RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8600.142801] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8600.144594] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8600.146853] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8600.149601] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8600.151846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8600.153453] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8600.155207] Call Trace: [ 8600.156083] [ 8600.156588] ? show_regs.cold.9+0x22/0x2f [ 8600.158084] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8600.159417] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8600.160764] ? watchdog+0x40/0x40 [ 8600.161618] ? hrtimer_interrupt+0x128/0x2c0 [ 8600.163103] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8600.164488] ? apic_timer_interrupt+0xf/0x20 [ 8600.165675] [ 8600.166452] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8600.167834] do_raw_spin_lock+0x177/0x190 [ 8600.169177] _raw_spin_lock+0x21/0x30 [ 8600.170595] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8600.172436] ? do_raw_spin_unlock+0x75/0x190 [ 8600.173651] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8600.176267] ? do_wait_intr+0xf0/0xf0 [ 8600.177860] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8600.179843] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8600.182818] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8600.185518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8600.188165] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8600.190624] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8600.192582] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8600.194474] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8600.196804] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8600.199402] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8600.201572] lmv_intent_open+0x338/0x1150 [lmv] [ 8600.203487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8600.205551] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8600.208594] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8600.211662] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8600.213902] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8600.216445] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8600.218419] ? do_raw_spin_unlock+0x75/0x190 [ 8600.220417] ? _raw_spin_unlock+0x12/0x30 [ 8600.222193] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8600.224563] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8600.226780] ll_atomic_open+0x36a/0xd70 [lustre] [ 8600.229571] lookup_open+0x356/0xb40 [ 8600.231266] do_last+0x313/0xfc0 [ 8600.232966] ? nd_jump_root+0xe5/0x160 [ 8600.234858] ? path_init+0x437/0x520 [ 8600.236747] path_openat+0xf7/0x500 [ 8600.238873] ? do_raw_spin_unlock+0x75/0x190 [ 8600.241123] do_filp_open+0x99/0x140 [ 8600.242803] ? getname_flags+0x6e/0x330 [ 8600.244684] ? __check_object_size+0xff/0x256 [ 8600.248537] ? do_raw_spin_unlock+0x75/0x190 [ 8600.251243] ? _raw_spin_unlock+0x12/0x30 [ 8600.253282] do_sys_openat2+0x2b4/0x410 [ 8600.254738] do_sys_open+0x73/0xa0 [ 8600.256473] __x64_sys_openat+0x24/0x30 [ 8600.258592] do_syscall_64+0xc1/0x440 [ 8600.260341] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8600.263183] RIP: 0033:0x7fdc0afbf332 [ 8600.264595] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8600.277574] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8600.282497] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8600.286439] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8600.293398] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8600.299310] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8600.303138] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8608.048012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8608.051319] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8608.060179] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8608.072687] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8608.072701] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8608.072705] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8608.072719] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8608.072724] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8608.072729] RAX: 0000000000005afc RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8608.072732] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8608.072734] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8608.072740] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8608.072742] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8608.072746] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8608.072749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8608.072751] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8608.072759] Call Trace: [ 8608.072765] [ 8608.072770] ? show_regs.cold.9+0x22/0x2f [ 8608.072780] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8608.072792] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8608.072797] ? watchdog+0x40/0x40 [ 8608.072801] ? hrtimer_interrupt+0x128/0x2c0 [ 8608.072809] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8608.072822] ? apic_timer_interrupt+0xf/0x20 [ 8608.072827] [ 8608.072830] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8608.072838] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8608.072842] ? dump_stack+0xf6/0x10e [ 8608.072848] do_raw_spin_lock+0x177/0x190 [ 8608.072853] _raw_spin_lock+0x21/0x30 [ 8608.072861] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8608.076997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8608.077093] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8608.077113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8608.077116] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8608.077130] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8608.077134] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8608.077138] RAX: 0000000000000526 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8608.077141] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8608.077143] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8608.077145] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8608.077147] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8608.077150] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8608.077152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8608.077163] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8608.077170] Call Trace: [ 8608.077175] [ 8608.077181] ? show_regs.cold.9+0x22/0x2f [ 8608.077191] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8608.077197] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8608.077202] ? watchdog+0x40/0x40 [ 8608.077205] ? hrtimer_interrupt+0x128/0x2c0 [ 8608.077209] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8608.077353] ? apic_timer_interrupt+0xf/0x20 [ 8608.077359] [ 8608.077362] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8608.077369] do_raw_spin_lock+0x177/0x190 [ 8608.077458] _raw_spin_lock+0x21/0x30 [ 8608.077465] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8608.084492] obd_mod_free+0x46/0x140 [ptlrpc] [ 8608.085090] mdc_commit_open+0x91/0xb0 [mdc] [ 8608.085144] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8608.085413] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8608.085736] after_reply+0xe2c/0x1780 [ptlrpc] [ 8608.086316] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8608.086634] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8608.092205] ? do_raw_spin_unlock+0x75/0x190 [ 8608.092237] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8608.092549] ? do_wait_intr+0xf0/0xf0 [ 8608.092562] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8608.092877] mdc_close+0x53b/0xd70 [mdc] [ 8608.092962] lmv_close+0x18d/0x490 [lmv] [ 8608.093005] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8608.093142] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8608.093441] ll_md_close+0x274/0x800 [lustre] [ 8608.097453] ? do_wait_intr+0xf0/0xf0 [ 8608.097474] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8608.097982] kthread+0x1d1/0x200 [ 8608.097990] ? set_kthread_struct+0x70/0x70 [ 8608.097994] ret_from_fork+0x1f/0x30 [ 8608.353349] ll_file_release+0x128/0xc50 [lustre] [ 8608.355697] __fput+0x10d/0x440 [ 8608.356946] ____fput+0x12/0x20 [ 8608.358340] task_work_run+0xc6/0x110 [ 8608.359869] exit_to_usermode_loop+0x1dd/0x1f0 [ 8608.361404] do_syscall_64+0x3d6/0x440 [ 8608.363367] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8608.365335] RIP: 0033:0x7fdc0afbfcd8 [ 8608.366948] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8608.375278] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8608.380835] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8608.384541] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8608.388309] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8608.391023] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8608.394423] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8612.051022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8612.054976] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8612.077580] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8612.082884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8612.086733] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8612.090285] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8612.092271] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8612.101150] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8612.104434] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8612.107272] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8612.110598] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8612.113418] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8612.116367] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8612.119223] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8612.122832] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8612.125453] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8612.128617] Call Trace: [ 8612.130078] [ 8612.130983] ? show_regs.cold.9+0x22/0x2f [ 8612.132274] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8612.134533] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8612.136197] ? watchdog+0x40/0x40 [ 8612.137292] ? hrtimer_interrupt+0x128/0x2c0 [ 8612.138902] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8612.140992] ? apic_timer_interrupt+0xf/0x20 [ 8612.142560] [ 8612.143250] ? native_safe_halt+0x2e/0x50 [ 8612.145134] kvm_wait+0xcd/0xe0 [ 8612.146230] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8612.148573] do_raw_spin_lock+0x177/0x190 [ 8612.150094] _raw_spin_lock+0x21/0x30 [ 8612.151672] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8612.154121] process_one_work+0x2c8/0x700 [ 8612.155808] worker_thread+0x296/0x6e0 [ 8612.157740] ? rescuer_thread+0x570/0x570 [ 8612.159541] kthread+0x1d1/0x200 [ 8612.161177] ? set_kthread_struct+0x70/0x70 [ 8612.163065] ret_from_fork+0x1f/0x30 [ 8613.343484] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6588s! [ 8613.348413] Showing busy workqueues and worker pools: [ 8613.350180] workqueue events: flags=0x0 [ 8613.352185] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8613.355516] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8613.363619] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8613.371238] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8613.379260] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8613.387371] pending: kfree_rcu_monitor [ 8613.390040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8613.394306] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8613.403551] workqueue events_long: flags=0x0 [ 8613.407361] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8613.409960] pending: lu_objects_flush [obdclass] [ 8613.412102] workqueue events_unbound: flags=0x2 [ 8613.413611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8613.415495] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8613.418557] workqueue events_power_efficient: flags=0x80 [ 8613.419975] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8613.421991] pending: neigh_periodic_work, check_lifetime [ 8613.423526] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8613.425409] pending: do_cache_clean, neigh_periodic_work [ 8613.427179] workqueue mm_percpu_wq: flags=0x8 [ 8613.428850] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8613.431389] pending: lru_add_drain_per_cpu [ 8613.432862] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8613.435265] pending: vmstat_update, lru_add_drain_per_cpu [ 8613.437840] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8613.439916] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8613.441910] workqueue writeback: flags=0x4a [ 8613.443423] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8613.445266] pending: wb_workfn, wb_workfn [ 8613.446839] workqueue xprtiod: flags=0xa [ 8613.448585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8613.450795] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8613.453558] workqueue ipv6_addrconf: flags=0x40008 [ 8613.455011] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8613.457385] pending: addrconf_verify_work [ 8613.459321] workqueue knbd0-recv: flags=0x1a [ 8613.460814] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8613.462948] in-flight: 655:recv_work [ 8613.464266] workqueue ptlrpc_pinger: flags=0x2 [ 8613.465810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8613.467529] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8613.469074] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6588s workers=4 idle: 36396 38 11188 [ 8613.471786] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8604s workers=2 idle: 658 [ 8628.059582] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8628.069989] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8628.139551] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8628.156544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8628.168034] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8628.174753] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8628.193985] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8628.202472] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8628.207284] RDX: 000000000000441a RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8628.209246] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8628.218026] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8628.223640] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8628.227873] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8628.232968] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8628.236036] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8628.239518] Call Trace: [ 8628.240590] [ 8628.241618] ? show_regs.cold.9+0x22/0x2f [ 8628.243937] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8628.246056] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8628.248598] ? watchdog+0x40/0x40 [ 8628.250169] ? hrtimer_interrupt+0x128/0x2c0 [ 8628.252235] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8628.254524] ? apic_timer_interrupt+0xf/0x20 [ 8628.256742] [ 8628.257788] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8628.260323] do_raw_spin_lock+0x177/0x190 [ 8628.262173] _raw_spin_lock+0x21/0x30 [ 8628.263853] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8628.267553] ? do_raw_spin_unlock+0x75/0x190 [ 8628.268851] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8628.272582] ? do_wait_intr+0xf0/0xf0 [ 8628.275957] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8628.279348] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8628.283069] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8628.286837] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8628.289310] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8628.292176] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8628.295047] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8628.297153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8628.299960] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8628.302564] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8628.304883] lmv_intent_open+0x338/0x1150 [lmv] [ 8628.307025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8628.309502] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8628.312274] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8628.314528] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8628.316880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8628.318996] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8628.322142] ? do_raw_spin_unlock+0x75/0x190 [ 8628.324971] ? _raw_spin_unlock+0x12/0x30 [ 8628.326862] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8628.329515] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8628.332500] ll_atomic_open+0x36a/0xd70 [lustre] [ 8628.335278] lookup_open+0x356/0xb40 [ 8628.337535] do_last+0x313/0xfc0 [ 8628.339826] ? nd_jump_root+0xe5/0x160 [ 8628.341490] ? path_init+0x437/0x520 [ 8628.342855] path_openat+0xf7/0x500 [ 8628.347085] ? do_raw_spin_unlock+0x75/0x190 [ 8628.351062] do_filp_open+0x99/0x140 [ 8628.352984] ? getname_flags+0x6e/0x330 [ 8628.355572] ? __check_object_size+0xff/0x256 [ 8628.357974] ? do_raw_spin_unlock+0x75/0x190 [ 8628.360223] ? _raw_spin_unlock+0x12/0x30 [ 8628.361974] do_sys_openat2+0x2b4/0x410 [ 8628.363587] do_sys_open+0x73/0xa0 [ 8628.365845] __x64_sys_openat+0x24/0x30 [ 8628.368249] do_syscall_64+0xc1/0x440 [ 8628.375557] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8628.379306] RIP: 0033:0x7fdc0afbf332 [ 8628.386628] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8628.419300] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8628.432316] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8628.440302] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8628.447517] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8628.458839] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8628.472206] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8636.054034] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8636.057362] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8636.077638] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8636.082387] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8636.085854] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8636.088615] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8636.096114] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8636.099164] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8636.102446] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8636.105593] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8636.108439] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8636.111774] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8636.115000] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8636.118632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8636.121310] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8636.124027] Call Trace: [ 8636.125079] [ 8636.125963] ? show_regs.cold.9+0x22/0x2f [ 8636.127594] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8636.129425] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8636.131029] ? watchdog+0x40/0x40 [ 8636.132396] ? hrtimer_interrupt+0x128/0x2c0 [ 8636.134087] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8636.135910] ? apic_timer_interrupt+0xf/0x20 [ 8636.137613] [ 8636.138698] ? native_safe_halt+0x2e/0x50 [ 8636.140228] kvm_wait+0xcd/0xe0 [ 8636.141534] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8636.143505] do_raw_spin_lock+0x177/0x190 [ 8636.145187] _raw_spin_lock+0x21/0x30 [ 8636.146728] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8636.149511] ? do_raw_spin_unlock+0x75/0x190 [ 8636.151070] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8636.153417] ? do_wait_intr+0xf0/0xf0 [ 8636.155159] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8636.157035] mdc_close+0x53b/0xd70 [mdc] [ 8636.158692] lmv_close+0x18d/0x490 [lmv] [ 8636.160155] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8636.162729] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8636.164724] ll_md_close+0x274/0x800 [lustre] [ 8636.166685] ll_file_release+0x128/0xc50 [lustre] [ 8636.168800] __fput+0x10d/0x440 [ 8636.169813] ____fput+0x12/0x20 [ 8636.171055] task_work_run+0xc6/0x110 [ 8636.172797] exit_to_usermode_loop+0x1dd/0x1f0 [ 8636.174538] do_syscall_64+0x3d6/0x440 [ 8636.176070] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8636.178108] RIP: 0033:0x7fdc0afbfcd8 [ 8636.179617] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8636.187508] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8636.190569] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8636.193974] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8636.196567] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8636.198985] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8636.201486] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8640.048028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8640.051020] Modules linked in: lustre(O) [ 8640.051026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8640.051037] Modules linked in: [ 8640.052273] osp(O) [ 8640.054411] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8640.054501] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8640.054512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8640.054519] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8640.055881] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8640.055953] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8640.055960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8640.055963] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8640.055976] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8640.055980] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8640.055985] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8640.055988] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8640.055990] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8640.055992] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8640.055995] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8640.055999] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8640.056002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8640.056004] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8640.056013] Call Trace: [ 8640.056019] [ 8640.056024] ? show_regs.cold.9+0x22/0x2f [ 8640.056035] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8640.056041] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8640.056047] ? watchdog+0x40/0x40 [ 8640.056052] ? hrtimer_interrupt+0x128/0x2c0 [ 8640.056057] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8640.056064] ? apic_timer_interrupt+0xf/0x20 [ 8640.056070] [ 8640.056073] ? native_safe_halt+0x2e/0x50 [ 8640.056078] kvm_wait+0xcd/0xe0 [ 8640.056087] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8640.056094] ? dump_stack+0xf6/0x10e [ 8640.056099] do_raw_spin_lock+0x177/0x190 [ 8640.056103] _raw_spin_lock+0x21/0x30 [ 8640.056109] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8640.056369] obd_mod_free+0x46/0x140 [ptlrpc] [ 8640.056564] mdc_commit_open+0x91/0xb0 [mdc] [ 8640.056615] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8640.056815] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8640.057017] after_reply+0xe2c/0x1780 [ptlrpc] [ 8640.057224] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8640.058297] [ 8640.058302] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8640.058315] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8640.058318] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8640.058321] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8640.058323] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8640.058325] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8640.058326] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8640.058330] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8640.058333] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8640.058335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8640.058336] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8640.058342] Call Trace: [ 8640.058345] [ 8640.058348] ? show_regs.cold.9+0x22/0x2f [ 8640.058355] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8640.058359] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8640.058362] ? watchdog+0x40/0x40 [ 8640.058364] ? hrtimer_interrupt+0x128/0x2c0 [ 8640.058367] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8640.058372] ? apic_timer_interrupt+0xf/0x20 [ 8640.058375] [ 8640.058377] ? native_safe_halt+0x2e/0x50 [ 8640.058379] kvm_wait+0xcd/0xe0 [ 8640.058385] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8640.058392] do_raw_spin_lock+0x177/0x190 [ 8640.058395] _raw_spin_lock+0x21/0x30 [ 8640.058398] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8640.058591] process_one_work+0x2c8/0x700 [ 8640.058607] worker_thread+0x296/0x6e0 [ 8640.058612] ? rescuer_thread+0x570/0x570 [ 8640.058616] kthread+0x1d1/0x200 [ 8640.058620] ? set_kthread_struct+0x70/0x70 [ 8640.058623] ret_from_fork+0x1f/0x30 [ 8640.451157] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8640.453042] ? do_wait_intr+0xf0/0xf0 [ 8640.454512] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8640.457116] kthread+0x1d1/0x200 [ 8640.460778] ? set_kthread_struct+0x70/0x70 [ 8640.463989] ret_from_fork+0x1f/0x30 [ 8643.551327] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6618s! [ 8643.559451] Showing busy workqueues and worker pools: [ 8643.561441] workqueue events: flags=0x0 [ 8643.563496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8643.565957] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8643.568526] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8643.570992] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8643.574944] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8643.577446] pending: kfree_rcu_monitor [ 8643.578886] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8643.581345] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8643.588088] workqueue events_long: flags=0x0 [ 8643.589980] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8643.592921] pending: lu_objects_flush [obdclass] [ 8643.595641] workqueue events_unbound: flags=0x2 [ 8643.597620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8643.601084] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8643.606062] workqueue events_power_efficient: flags=0x80 [ 8643.608811] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8643.611233] pending: neigh_periodic_work, check_lifetime [ 8643.613644] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8643.616234] pending: do_cache_clean, neigh_periodic_work [ 8643.619467] workqueue mm_percpu_wq: flags=0x8 [ 8643.621539] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8643.624349] pending: lru_add_drain_per_cpu [ 8643.626094] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8643.628694] pending: vmstat_update, lru_add_drain_per_cpu [ 8643.630835] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8643.633565] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8643.636509] workqueue writeback: flags=0x4a [ 8643.637953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8643.640508] pending: wb_workfn, wb_workfn [ 8643.642202] workqueue xprtiod: flags=0xa [ 8643.643835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8643.646293] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8643.649591] workqueue ipv6_addrconf: flags=0x40008 [ 8643.651464] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8643.654019] pending: addrconf_verify_work [ 8643.655413] workqueue knbd0-recv: flags=0x1a [ 8643.656743] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8643.659032] in-flight: 655:recv_work [ 8643.660423] workqueue ptlrpc_pinger: flags=0x2 [ 8643.661864] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8643.664238] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8643.666713] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6618s workers=4 idle: 36396 38 11188 [ 8643.669965] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8634s workers=2 idle: 658 [ 8656.052015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8656.055782] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8656.077161] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8656.082707] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8656.087065] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8656.089909] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8656.099803] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8656.103903] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8656.108291] RDX: 00000000000048ef RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8656.111899] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8656.115857] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8656.119702] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8656.124272] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8656.128426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8656.131823] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8656.135014] Call Trace: [ 8656.136122] [ 8656.137098] ? show_regs.cold.9+0x22/0x2f [ 8656.139118] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8656.141134] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8656.143371] ? watchdog+0x40/0x40 [ 8656.144521] ? hrtimer_interrupt+0x128/0x2c0 [ 8656.146023] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8656.152431] ? apic_timer_interrupt+0xf/0x20 [ 8656.154241] [ 8656.158203] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8656.160625] do_raw_spin_lock+0x177/0x190 [ 8656.161865] _raw_spin_lock+0x21/0x30 [ 8656.166818] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8656.168901] ? do_raw_spin_unlock+0x75/0x190 [ 8656.174701] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8656.176733] ? do_wait_intr+0xf0/0xf0 [ 8656.181118] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8656.184330] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8656.189083] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8656.194371] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8656.197582] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8656.201825] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8656.203898] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8656.208510] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8656.211789] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8656.221332] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8656.223167] lmv_intent_open+0x338/0x1150 [lmv] [ 8656.224872] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8656.226835] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8656.229240] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8656.231278] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8656.233541] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8656.235739] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8656.237674] ? do_raw_spin_unlock+0x75/0x190 [ 8656.239370] ? _raw_spin_unlock+0x12/0x30 [ 8656.241007] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8656.245524] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8656.247449] ll_atomic_open+0x36a/0xd70 [lustre] [ 8656.249749] lookup_open+0x356/0xb40 [ 8656.251248] do_last+0x313/0xfc0 [ 8656.255435] ? nd_jump_root+0xe5/0x160 [ 8656.259877] ? path_init+0x437/0x520 [ 8656.263781] path_openat+0xf7/0x500 [ 8656.268645] ? do_raw_spin_unlock+0x75/0x190 [ 8656.270148] do_filp_open+0x99/0x140 [ 8656.271282] ? getname_flags+0x6e/0x330 [ 8656.277821] ? __check_object_size+0xff/0x256 [ 8656.279958] ? do_raw_spin_unlock+0x75/0x190 [ 8656.284328] ? _raw_spin_unlock+0x12/0x30 [ 8656.286008] do_sys_openat2+0x2b4/0x410 [ 8656.287670] do_sys_open+0x73/0xa0 [ 8656.288868] __x64_sys_openat+0x24/0x30 [ 8656.291651] do_syscall_64+0xc1/0x440 [ 8656.293367] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8656.294626] RIP: 0033:0x7fdc0afbf332 [ 8656.299258] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8656.320199] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8656.332653] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8656.343796] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8656.350099] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8656.355115] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8656.358193] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8664.056049] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 8664.063748] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8664.102446] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8664.106855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8664.110506] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8664.112660] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8664.120185] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8664.123713] RAX: 0000000000001f8a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8664.126700] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8664.129577] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8664.132671] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8664.135243] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8664.138327] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8664.142564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8664.145722] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8664.148950] Call Trace: [ 8664.150892] [ 8664.151865] ? show_regs.cold.9+0x22/0x2f [ 8664.153222] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8664.155213] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8664.157165] ? watchdog+0x40/0x40 [ 8664.159067] ? hrtimer_interrupt+0x128/0x2c0 [ 8664.161076] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8664.163269] ? apic_timer_interrupt+0xf/0x20 [ 8664.165192] [ 8664.166272] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8664.168971] do_raw_spin_lock+0x177/0x190 [ 8664.170680] _raw_spin_lock+0x21/0x30 [ 8664.172437] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8664.175103] ? do_raw_spin_unlock+0x75/0x190 [ 8664.177345] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8664.179667] ? do_wait_intr+0xf0/0xf0 [ 8664.181228] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8664.184487] mdc_close+0x53b/0xd70 [mdc] [ 8664.186431] lmv_close+0x18d/0x490 [lmv] [ 8664.188433] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8664.191570] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8664.194111] ll_md_close+0x274/0x800 [lustre] [ 8664.196003] ll_file_release+0x128/0xc50 [lustre] [ 8664.198437] __fput+0x10d/0x440 [ 8664.199740] ____fput+0x12/0x20 [ 8664.200989] task_work_run+0xc6/0x110 [ 8664.202963] exit_to_usermode_loop+0x1dd/0x1f0 [ 8664.204854] do_syscall_64+0x3d6/0x440 [ 8664.206775] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8664.208676] RIP: 0033:0x7fdc0afbfcd8 [ 8664.210391] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8664.218984] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8664.222161] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8664.225581] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8664.228414] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8664.231057] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8664.234773] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8668.050079] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8668.054025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8668.059519] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8668.059600] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8668.059608] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8668.059611] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8668.059625] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8668.059629] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8668.059634] RAX: 0000000000006f83 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8668.059636] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8668.059638] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8668.059640] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8668.059643] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8668.059646] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8668.059648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8668.059651] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8668.059660] Call Trace: [ 8668.059665] [ 8668.059671] ? show_regs.cold.9+0x22/0x2f [ 8668.064705] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8668.064787] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8668.064793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8668.064800] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8668.065074] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8668.065084] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8668.065088] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8668.065092] RAX: 000000000000689f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8668.065095] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8668.065097] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8668.065099] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8668.065101] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8668.065122] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8668.065126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8668.065129] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8668.065136] Call Trace: [ 8668.065141] [ 8668.065146] ? show_regs.cold.9+0x22/0x2f [ 8668.065155] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8668.065161] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8668.065165] ? watchdog+0x40/0x40 [ 8668.065170] ? hrtimer_interrupt+0x128/0x2c0 [ 8668.065174] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8668.065181] ? apic_timer_interrupt+0xf/0x20 [ 8668.065186] [ 8668.065188] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8668.065193] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8668.065198] do_raw_spin_lock+0x177/0x190 [ 8668.065203] _raw_spin_lock+0x21/0x30 [ 8668.065207] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8668.065428] process_one_work+0x2c8/0x700 [ 8668.065435] worker_thread+0x296/0x6e0 [ 8668.065439] ? rescuer_thread+0x570/0x570 [ 8668.065443] kthread+0x1d1/0x200 [ 8668.065448] ? set_kthread_struct+0x70/0x70 [ 8668.065452] ret_from_fork+0x1f/0x30 [ 8668.231518] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8668.233513] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8668.235416] ? watchdog+0x40/0x40 [ 8668.236521] ? hrtimer_interrupt+0x128/0x2c0 [ 8668.238375] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8668.239949] ? apic_timer_interrupt+0xf/0x20 [ 8668.241178] [ 8668.242007] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8668.243973] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8668.246045] ? dump_stack+0xf6/0x10e [ 8668.247175] do_raw_spin_lock+0x177/0x190 [ 8668.248526] _raw_spin_lock+0x21/0x30 [ 8668.249970] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8668.251902] obd_mod_free+0x46/0x140 [ptlrpc] [ 8668.253887] mdc_commit_open+0x91/0xb0 [mdc] [ 8668.255171] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8668.257235] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8668.259306] after_reply+0xe2c/0x1780 [ptlrpc] [ 8668.261119] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8668.263166] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8668.264804] ? do_wait_intr+0xf0/0xf0 [ 8668.265900] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8668.267892] kthread+0x1d1/0x200 [ 8668.269081] ? set_kthread_struct+0x70/0x70 [ 8668.270392] ret_from_fork+0x1f/0x30 [ 8673.759369] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6648s! [ 8673.773329] Showing busy workqueues and worker pools: [ 8673.779481] workqueue events: flags=0x0 [ 8673.781167] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8673.784282] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8673.787286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8673.791105] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8673.796951] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8673.802229] pending: kfree_rcu_monitor [ 8673.804435] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8673.808448] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8673.815406] workqueue events_long: flags=0x0 [ 8673.817198] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8673.820061] pending: lu_objects_flush [obdclass] [ 8673.823091] workqueue events_unbound: flags=0x2 [ 8673.825249] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8673.828206] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8673.833288] workqueue events_power_efficient: flags=0x80 [ 8673.835533] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8673.838670] pending: neigh_periodic_work, check_lifetime [ 8673.842378] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8673.851048] pending: do_cache_clean, neigh_periodic_work [ 8673.858288] workqueue mm_percpu_wq: flags=0x8 [ 8673.860415] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8673.864220] pending: lru_add_drain_per_cpu [ 8673.865299] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8673.867779] pending: vmstat_update, lru_add_drain_per_cpu [ 8673.870388] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8673.876294] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8673.881210] workqueue writeback: flags=0x4a [ 8673.884047] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8673.888040] pending: wb_workfn, wb_workfn [ 8673.889919] workqueue xprtiod: flags=0xa [ 8673.892376] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8673.896124] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8673.899226] workqueue ipv6_addrconf: flags=0x40008 [ 8673.902118] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8673.906578] pending: addrconf_verify_work [ 8673.908297] workqueue knbd0-recv: flags=0x1a [ 8673.910158] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8673.911909] in-flight: 655:recv_work [ 8673.914433] workqueue ptlrpc_pinger: flags=0x2 [ 8673.917021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8673.920117] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8673.922992] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6648s workers=4 idle: 36396 38 11188 [ 8673.927838] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8664s workers=2 idle: 658 [ 8684.054041] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8684.058297] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8684.087515] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8684.093800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8684.100176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8684.106480] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8684.120229] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8684.129030] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8684.135847] RDX: 00000000000000ac RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8684.140114] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8684.147486] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8684.150809] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8684.156408] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8684.167866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8684.174407] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8684.182956] Call Trace: [ 8684.186753] [ 8684.187845] ? show_regs.cold.9+0x22/0x2f [ 8684.190009] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8684.193045] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8684.198237] ? watchdog+0x40/0x40 [ 8684.199758] ? hrtimer_interrupt+0x128/0x2c0 [ 8684.201304] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8684.205517] ? apic_timer_interrupt+0xf/0x20 [ 8684.209874] [ 8684.211324] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8684.223557] do_raw_spin_lock+0x177/0x190 [ 8684.232096] _raw_spin_lock+0x21/0x30 [ 8684.238886] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8684.246809] ? do_raw_spin_unlock+0x75/0x190 [ 8684.250778] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8684.260221] ? do_wait_intr+0xf0/0xf0 [ 8684.268515] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8684.279883] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8684.291372] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8684.297101] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8684.305795] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8684.312174] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8684.317958] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8684.320133] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8684.324634] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8684.335843] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8684.337850] lmv_intent_open+0x338/0x1150 [lmv] [ 8684.344938] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8684.346426] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8684.350430] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8684.351849] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8684.353213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8684.363851] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8684.368748] ? do_raw_spin_unlock+0x75/0x190 [ 8684.370562] ? _raw_spin_unlock+0x12/0x30 [ 8684.374150] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8684.377616] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8684.380100] ll_atomic_open+0x36a/0xd70 [lustre] [ 8684.384889] lookup_open+0x356/0xb40 [ 8684.387024] do_last+0x313/0xfc0 [ 8684.389850] ? nd_jump_root+0xe5/0x160 [ 8684.391525] ? path_init+0x437/0x520 [ 8684.392646] path_openat+0xf7/0x500 [ 8684.395404] ? do_raw_spin_unlock+0x75/0x190 [ 8684.398824] do_filp_open+0x99/0x140 [ 8684.401868] ? getname_flags+0x6e/0x330 [ 8684.403409] ? __check_object_size+0xff/0x256 [ 8684.409976] ? do_raw_spin_unlock+0x75/0x190 [ 8684.412124] ? _raw_spin_unlock+0x12/0x30 [ 8684.415159] do_sys_openat2+0x2b4/0x410 [ 8684.418531] do_sys_open+0x73/0xa0 [ 8684.419953] __x64_sys_openat+0x24/0x30 [ 8684.421917] do_syscall_64+0xc1/0x440 [ 8684.424187] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8684.426100] RIP: 0033:0x7fdc0afbf332 [ 8684.427559] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8684.449788] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8684.454256] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8684.461401] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8684.468282] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8684.474409] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8684.480335] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8692.054015] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 8692.058758] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8692.085921] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8692.091247] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8692.096794] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8692.099985] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8692.110992] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8692.114693] RAX: 00000000000069f1 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8692.117807] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8692.122234] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8692.126225] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8692.130597] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8692.138787] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8692.147155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8692.152757] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8692.160106] Call Trace: [ 8692.161897] [ 8692.163128] ? show_regs.cold.9+0x22/0x2f [ 8692.166214] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8692.170831] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8692.176614] ? watchdog+0x40/0x40 [ 8692.178262] ? hrtimer_interrupt+0x128/0x2c0 [ 8692.184242] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8692.186285] ? apic_timer_interrupt+0xf/0x20 [ 8692.193536] [ 8692.194287] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8692.200598] do_raw_spin_lock+0x177/0x190 [ 8692.204638] _raw_spin_lock+0x21/0x30 [ 8692.206696] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8692.208641] ? do_raw_spin_unlock+0x75/0x190 [ 8692.211764] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8692.215169] ? do_wait_intr+0xf0/0xf0 [ 8692.217198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8692.220040] mdc_close+0x53b/0xd70 [mdc] [ 8692.222121] lmv_close+0x18d/0x490 [lmv] [ 8692.224049] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8692.227106] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8692.230504] ll_md_close+0x274/0x800 [lustre] [ 8692.232934] ll_file_release+0x128/0xc50 [lustre] [ 8692.235495] __fput+0x10d/0x440 [ 8692.236975] ____fput+0x12/0x20 [ 8692.238356] task_work_run+0xc6/0x110 [ 8692.240023] exit_to_usermode_loop+0x1dd/0x1f0 [ 8692.242694] do_syscall_64+0x3d6/0x440 [ 8692.244304] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8692.246935] RIP: 0033:0x7fdc0afbfcd8 [ 8692.248770] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8692.257588] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8692.261036] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8692.264111] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8692.267625] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8692.271138] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8692.275667] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8696.052631] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8696.056897] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8696.078772] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8696.083071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8696.089390] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8696.093745] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8696.098715] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8696.114018] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8696.121063] RAX: 0000000000007090 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8696.126375] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8696.130286] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8696.133495] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8696.136438] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8696.140561] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8696.145403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8696.147150] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8696.149653] Call Trace: [ 8696.150632] [ 8696.151189] ? show_regs.cold.9+0x22/0x2f [ 8696.154580] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8696.161059] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8696.163262] ? watchdog+0x40/0x40 [ 8696.164541] ? hrtimer_interrupt+0x128/0x2c0 [ 8696.165586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8696.167657] ? apic_timer_interrupt+0xf/0x20 [ 8696.169560] [ 8696.170317] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8696.172232] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8696.174280] do_raw_spin_lock+0x177/0x190 [ 8696.175903] _raw_spin_lock+0x21/0x30 [ 8696.177351] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8696.179858] process_one_work+0x2c8/0x700 [ 8696.181440] worker_thread+0x296/0x6e0 [ 8696.182558] ? rescuer_thread+0x570/0x570 [ 8696.183743] kthread+0x1d1/0x200 [ 8696.184787] ? set_kthread_struct+0x70/0x70 [ 8696.186066] ret_from_fork+0x1f/0x30 [ 8700.049426] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8700.053023] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8700.104229] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8700.110065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8700.122720] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8700.124315] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8700.138082] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8700.143208] RAX: 0000000000007f87 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8700.150078] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8700.153643] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8700.155485] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8700.163268] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8700.167069] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8700.175946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8700.179359] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8700.181214] Call Trace: [ 8700.184928] [ 8700.187412] ? show_regs.cold.9+0x22/0x2f [ 8700.191035] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8700.196114] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8700.199990] ? watchdog+0x40/0x40 [ 8700.204010] ? hrtimer_interrupt+0x128/0x2c0 [ 8700.207821] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8700.210146] ? apic_timer_interrupt+0xf/0x20 [ 8700.217382] [ 8700.219486] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8700.224063] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8700.229732] ? dump_stack+0xf6/0x10e [ 8700.231479] do_raw_spin_lock+0x177/0x190 [ 8700.233940] _raw_spin_lock+0x21/0x30 [ 8700.236235] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8700.239340] obd_mod_free+0x46/0x140 [ptlrpc] [ 8700.242949] mdc_commit_open+0x91/0xb0 [mdc] [ 8700.245230] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8700.248410] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8700.255131] after_reply+0xe2c/0x1780 [ptlrpc] [ 8700.258075] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8700.261715] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8700.264473] ? do_wait_intr+0xf0/0xf0 [ 8700.266831] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8700.270891] kthread+0x1d1/0x200 [ 8700.272235] ? set_kthread_struct+0x70/0x70 [ 8700.274422] ret_from_fork+0x1f/0x30 [ 8703.967078] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6679s! [ 8703.975425] Showing busy workqueues and worker pools: [ 8703.979824] workqueue events: flags=0x0 [ 8703.982046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8703.988954] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8703.994642] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8703.998908] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8704.009818] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8704.017592] pending: kfree_rcu_monitor [ 8704.020020] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8704.026499] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8704.037965] workqueue events_long: flags=0x0 [ 8704.039756] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8704.042866] pending: lu_objects_flush [obdclass] [ 8704.045365] workqueue events_unbound: flags=0x2 [ 8704.054838] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8704.061631] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8704.066710] workqueue events_power_efficient: flags=0x80 [ 8704.072423] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8704.075633] pending: neigh_periodic_work, check_lifetime [ 8704.077943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8704.084848] pending: do_cache_clean, neigh_periodic_work [ 8704.088383] workqueue mm_percpu_wq: flags=0x8 [ 8704.090775] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8704.095303] pending: lru_add_drain_per_cpu [ 8704.097497] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8704.102012] pending: vmstat_update, lru_add_drain_per_cpu [ 8704.103850] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8704.115778] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8704.120726] workqueue writeback: flags=0x4a [ 8704.122827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8704.127043] pending: wb_workfn, wb_workfn [ 8704.129017] workqueue xprtiod: flags=0xa [ 8704.130898] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8704.133773] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8704.137461] workqueue ipv6_addrconf: flags=0x40008 [ 8704.139535] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8704.144990] pending: addrconf_verify_work [ 8704.147040] workqueue knbd0-recv: flags=0x1a [ 8704.148977] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8704.151855] in-flight: 655:recv_work [ 8704.155017] workqueue ptlrpc_pinger: flags=0x2 [ 8704.156871] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8704.159001] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8704.167597] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6679s workers=4 idle: 36396 38 11188 [ 8704.175445] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8694s workers=2 idle: 658 [ 8712.052067] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8712.055096] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8712.096970] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8712.106570] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8712.119566] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8712.124256] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8712.146473] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8712.155718] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8712.163470] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8712.165647] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8712.171907] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8712.177201] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8712.184663] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8712.189230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8712.191807] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8712.200204] Call Trace: [ 8712.201635] [ 8712.202978] ? show_regs.cold.9+0x22/0x2f [ 8712.204238] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8712.205230] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8712.206281] ? watchdog+0x40/0x40 [ 8712.206987] ? hrtimer_interrupt+0x128/0x2c0 [ 8712.213713] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8712.215886] ? apic_timer_interrupt+0xf/0x20 [ 8712.217823] [ 8712.218474] ? native_safe_halt+0x2e/0x50 [ 8712.219269] kvm_wait+0xcd/0xe0 [ 8712.220034] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8712.223616] do_raw_spin_lock+0x177/0x190 [ 8712.225704] _raw_spin_lock+0x21/0x30 [ 8712.227896] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8712.232536] ? do_raw_spin_unlock+0x75/0x190 [ 8712.235344] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8712.238588] ? do_wait_intr+0xf0/0xf0 [ 8712.239799] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8712.242269] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8712.244046] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8712.250622] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8712.254452] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8712.258181] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8712.261947] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8712.263546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8712.266817] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8712.270652] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8712.272775] lmv_intent_open+0x338/0x1150 [lmv] [ 8712.282193] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8712.290220] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8712.297722] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8712.302205] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8712.303799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8712.307128] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8712.311261] ? do_raw_spin_unlock+0x75/0x190 [ 8712.313856] ? _raw_spin_unlock+0x12/0x30 [ 8712.316151] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8712.320580] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8712.323067] ll_atomic_open+0x36a/0xd70 [lustre] [ 8712.325452] lookup_open+0x356/0xb40 [ 8712.327588] do_last+0x313/0xfc0 [ 8712.329167] ? nd_jump_root+0xe5/0x160 [ 8712.330899] ? path_init+0x437/0x520 [ 8712.336605] path_openat+0xf7/0x500 [ 8712.339206] ? do_raw_spin_unlock+0x75/0x190 [ 8712.343176] do_filp_open+0x99/0x140 [ 8712.346109] ? getname_flags+0x6e/0x330 [ 8712.347743] ? __check_object_size+0xff/0x256 [ 8712.349520] ? do_raw_spin_unlock+0x75/0x190 [ 8712.351216] ? _raw_spin_unlock+0x12/0x30 [ 8712.352875] do_sys_openat2+0x2b4/0x410 [ 8712.354247] do_sys_open+0x73/0xa0 [ 8712.356537] __x64_sys_openat+0x24/0x30 [ 8712.358486] do_syscall_64+0xc1/0x440 [ 8712.359765] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8712.361754] RIP: 0033:0x7fdc0afbf332 [ 8712.363235] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8712.373529] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8712.381161] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8712.387198] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8712.394641] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8712.400009] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8712.402688] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8720.054045] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 8720.061850] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8720.091163] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8720.095512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8720.099882] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8720.101429] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8720.109010] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8720.111556] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8720.113897] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8720.116734] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8720.119060] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8720.121718] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8720.123881] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8720.126264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8720.128037] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8720.135600] Call Trace: [ 8720.136398] [ 8720.137410] ? show_regs.cold.9+0x22/0x2f [ 8720.145003] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8720.148928] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8720.152438] ? watchdog+0x40/0x40 [ 8720.153683] ? hrtimer_interrupt+0x128/0x2c0 [ 8720.155315] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8720.160950] ? apic_timer_interrupt+0xf/0x20 [ 8720.163434] [ 8720.166517] ? native_safe_halt+0x2e/0x50 [ 8720.169279] kvm_wait+0xcd/0xe0 [ 8720.171297] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8720.173746] do_raw_spin_lock+0x177/0x190 [ 8720.176547] _raw_spin_lock+0x21/0x30 [ 8720.178772] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8720.184946] ? do_raw_spin_unlock+0x75/0x190 [ 8720.188327] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8720.192377] ? do_wait_intr+0xf0/0xf0 [ 8720.194418] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8720.197321] mdc_close+0x53b/0xd70 [mdc] [ 8720.198750] lmv_close+0x18d/0x490 [lmv] [ 8720.199861] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8720.202340] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8720.204479] ll_md_close+0x274/0x800 [lustre] [ 8720.205947] ll_file_release+0x128/0xc50 [lustre] [ 8720.209407] __fput+0x10d/0x440 [ 8720.211086] ____fput+0x12/0x20 [ 8720.213957] task_work_run+0xc6/0x110 [ 8720.217473] exit_to_usermode_loop+0x1dd/0x1f0 [ 8720.221585] do_syscall_64+0x3d6/0x440 [ 8720.222655] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8720.225040] RIP: 0033:0x7fdc0afbfcd8 [ 8720.226389] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8720.236896] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8720.245851] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8720.248802] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8720.252392] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8720.254955] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8720.257317] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8724.051011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8724.057220] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8724.092784] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8724.100658] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8724.107215] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8724.113304] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8724.117650] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8724.130530] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8724.134337] RAX: 0000000000005783 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8724.139665] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8724.144311] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8724.148114] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8724.151698] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8724.156023] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8724.160264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8724.163318] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8724.167061] Call Trace: [ 8724.168238] [ 8724.169328] ? show_regs.cold.9+0x22/0x2f [ 8724.170773] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8724.173081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8724.174759] ? watchdog+0x40/0x40 [ 8724.176004] ? hrtimer_interrupt+0x128/0x2c0 [ 8724.177906] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8724.179299] ? apic_timer_interrupt+0xf/0x20 [ 8724.180897] [ 8724.181660] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8724.183950] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8724.186345] do_raw_spin_lock+0x177/0x190 [ 8724.188737] _raw_spin_lock+0x21/0x30 [ 8724.189961] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8724.195013] process_one_work+0x2c8/0x700 [ 8724.197055] worker_thread+0x296/0x6e0 [ 8724.198665] ? rescuer_thread+0x570/0x570 [ 8724.201990] kthread+0x1d1/0x200 [ 8724.204098] ? set_kthread_struct+0x70/0x70 [ 8724.206242] ret_from_fork+0x1f/0x30 [ 8732.048025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8732.051715] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8732.075196] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8732.080246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8732.084531] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8732.086817] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8732.095458] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8732.098961] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8732.102360] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8732.106013] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8732.109005] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8732.112710] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8732.115766] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8732.119775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8732.122889] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8732.126432] Call Trace: [ 8732.127554] [ 8732.128361] ? show_regs.cold.9+0x22/0x2f [ 8732.129855] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8732.131843] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8732.133929] ? watchdog+0x40/0x40 [ 8732.135482] ? hrtimer_interrupt+0x128/0x2c0 [ 8732.137436] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8732.139754] ? apic_timer_interrupt+0xf/0x20 [ 8732.142435] [ 8732.143302] ? native_safe_halt+0x2e/0x50 [ 8732.146466] kvm_wait+0xcd/0xe0 [ 8732.148610] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8732.154096] ? dump_stack+0xf6/0x10e [ 8732.155615] do_raw_spin_lock+0x177/0x190 [ 8732.159318] _raw_spin_lock+0x21/0x30 [ 8732.161957] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8732.165311] obd_mod_free+0x46/0x140 [ptlrpc] [ 8732.169723] mdc_commit_open+0x91/0xb0 [mdc] [ 8732.175290] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8732.179255] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8732.182313] after_reply+0xe2c/0x1780 [ptlrpc] [ 8732.185300] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8732.188029] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8732.190127] ? do_wait_intr+0xf0/0xf0 [ 8732.192249] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8732.195153] kthread+0x1d1/0x200 [ 8732.196512] ? set_kthread_struct+0x70/0x70 [ 8732.198333] ret_from_fork+0x1f/0x30 [ 8734.687037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6709s! [ 8734.691211] Showing busy workqueues and worker pools: [ 8734.694850] workqueue events: flags=0x0 [ 8734.697373] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8734.701114] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8734.706016] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8734.709465] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8734.714987] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8734.717964] pending: kfree_rcu_monitor [ 8734.719838] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8734.723146] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8734.729367] workqueue events_long: flags=0x0 [ 8734.731102] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8734.734055] pending: lu_objects_flush [obdclass] [ 8734.736223] workqueue events_unbound: flags=0x2 [ 8734.737841] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8734.740173] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8734.744300] workqueue events_power_efficient: flags=0x80 [ 8734.746433] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8734.749116] pending: neigh_periodic_work, check_lifetime [ 8734.751278] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8734.753869] pending: do_cache_clean, neigh_periodic_work [ 8734.756587] workqueue mm_percpu_wq: flags=0x8 [ 8734.758220] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8734.760821] pending: lru_add_drain_per_cpu [ 8734.762489] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8734.764594] pending: vmstat_update, lru_add_drain_per_cpu [ 8734.766849] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8734.769621] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8734.772046] workqueue writeback: flags=0x4a [ 8734.773599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8734.776095] pending: wb_workfn, wb_workfn [ 8734.777972] workqueue xprtiod: flags=0xa [ 8734.779622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8734.782300] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8734.785703] workqueue ipv6_addrconf: flags=0x40008 [ 8734.787876] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8734.790418] pending: addrconf_verify_work [ 8734.792138] workqueue knbd0-recv: flags=0x1a [ 8734.794322] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8734.797071] in-flight: 655:recv_work [ 8734.798710] workqueue ptlrpc_pinger: flags=0x2 [ 8734.800616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8734.803135] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8734.805443] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6709s workers=4 idle: 36396 38 11188 [ 8734.808648] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8725s workers=2 idle: 658 [ 8739.504830] rcu: INFO: rcu_sched self-detected stall on CPU [ 8739.512206] rcu: 0-...!: (4221351 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 8739.524215] (t=6720161 jiffies g=270761 q=4288) [ 8739.531095] rcu: rcu_sched kthread starved for 6715093 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 8739.548560] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8739.562955] rcu: RCU grace-period kthread stack dump: [ 8739.564795] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8739.593206] Call Trace: [ 8739.594078] __schedule+0x351/0xcb0 [ 8739.595361] schedule+0xc0/0x180 [ 8739.606918] schedule_timeout+0xb4/0x190 [ 8739.608867] ? do_raw_spin_unlock+0x75/0x190 [ 8739.614601] ? __next_timer_interrupt+0x160/0x160 [ 8739.619181] ? prepare_to_swait_event+0xf3/0x140 [ 8739.623773] rcu_gp_kthread+0x8dd/0x15b0 [ 8739.628582] ? force_qs_rnp+0x2c0/0x2c0 [ 8739.632076] kthread+0x1d1/0x200 [ 8739.634082] ? set_kthread_struct+0x70/0x70 [ 8739.638406] ret_from_fork+0x1f/0x30 [ 8739.642170] rcu: Stack dump where RCU GP kthread last ran: [ 8739.651256] Sending NMI from CPU 0 to CPUs 3: [ 8739.681319] NMI backtrace for cpu 3 [ 8739.681327] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8739.681329] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8739.681339] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8739.681341] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8739.681343] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 [ 8739.681345] RAX: 0000000000006b90 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8739.681347] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8739.681348] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8739.681349] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8739.681351] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8739.681352] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8739.681353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8739.681355] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8739.681356] Call Trace: [ 8739.681356] [ 8739.681357] ? show_regs.cold.9+0x22/0x2f [ 8739.681358] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8739.681359] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8739.681360] ? nmi_handle+0xbd/0x210 [ 8739.681361] ? default_do_nmi+0x4b/0x1b0 [ 8739.681361] ? do_nmi+0x280/0x380 [ 8739.681362] ? end_repeat_nmi+0x16/0x1f [ 8739.681363] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8739.681364] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8739.681365] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8739.681366] [ 8739.681367] do_raw_spin_lock+0x177/0x190 [ 8739.681368] _raw_spin_lock+0x21/0x30 [ 8739.681368] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8739.681369] ? do_raw_spin_unlock+0x75/0x190 [ 8739.681370] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8739.681371] ? do_wait_intr+0xf0/0xf0 [ 8739.681372] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8739.681373] mdc_close+0x53b/0xd70 [mdc] [ 8739.681374] lmv_close+0x18d/0x490 [lmv] [ 8739.681374] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8739.681375] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8739.681376] ll_md_close+0x274/0x800 [lustre] [ 8739.681377] ll_file_release+0x128/0xc50 [lustre] [ 8739.681378] __fput+0x10d/0x440 [ 8739.681378] ____fput+0x12/0x20 [ 8739.681379] task_work_run+0xc6/0x110 [ 8739.681380] exit_to_usermode_loop+0x1dd/0x1f0 [ 8739.681381] do_syscall_64+0x3d6/0x440 [ 8739.681382] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8739.681382] RIP: 0033:0x7fdc0afbfcd8 [ 8739.681384] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8739.681385] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8739.681387] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8739.681388] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8739.681390] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8739.681391] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8739.681393] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8739.681398] NMI backtrace for cpu 0 [ 8739.675506] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 17.066 msecs [ 8739.706233] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8740.052111] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8740.066145] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8740.069143] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8740.069337] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8740.069343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8740.069346] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8740.069360] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8740.069363] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8740.069367] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8740.069370] RDX: 0000000000003fe0 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8740.069373] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8740.069375] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8740.069377] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8740.069381] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8740.069383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8740.069385] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8740.069393] Call Trace: [ 8740.069397] [ 8740.069402] ? show_regs.cold.9+0x22/0x2f [ 8740.069411] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8740.069417] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8740.069420] ? watchdog+0x40/0x40 [ 8740.069425] ? hrtimer_interrupt+0x128/0x2c0 [ 8740.069428] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8740.069436] ? apic_timer_interrupt+0xf/0x20 [ 8740.069441] [ 8740.069444] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8740.069448] do_raw_spin_lock+0x177/0x190 [ 8740.069453] _raw_spin_lock+0x21/0x30 [ 8740.069458] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8740.069733] ? do_raw_spin_unlock+0x75/0x190 [ 8740.069740] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8740.069938] ? do_wait_intr+0xf0/0xf0 [ 8740.069947] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8740.070211] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8740.070593] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8740.070889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8740.071082] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8740.071125] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8740.071153] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8740.071182] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8740.071268] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8740.078499] Call Trace: [ 8740.078514] [ 8740.078519] ? dump_stack+0xbb/0x10e [ 8740.078531] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8740.078538] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8740.078543] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8740.078548] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8740.078553] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8740.078557] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8740.078564] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8740.078569] ? update_process_times+0x6d/0xc0 [ 8740.082652] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8740.082706] lmv_intent_open+0x338/0x1150 [lmv] [ 8740.082733] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8740.082825] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8740.082915] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8740.082938] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8740.083092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8740.083332] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8740.083416] ? do_raw_spin_unlock+0x75/0x190 [ 8740.083425] ? _raw_spin_unlock+0x12/0x30 [ 8740.083432] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8740.083463] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8740.083478] ll_atomic_open+0x36a/0xd70 [lustre] [ 8740.083551] lookup_open+0x356/0xb40 [ 8740.083558] do_last+0x313/0xfc0 [ 8740.083562] ? nd_jump_root+0xe5/0x160 [ 8740.083565] ? path_init+0x437/0x520 [ 8740.083568] path_openat+0xf7/0x500 [ 8740.083572] ? do_raw_spin_unlock+0x75/0x190 [ 8740.083575] do_filp_open+0x99/0x140 [ 8740.083580] ? getname_flags+0x6e/0x330 [ 8740.083582] ? __check_object_size+0xff/0x256 [ 8740.083586] ? do_raw_spin_unlock+0x75/0x190 [ 8740.083589] ? _raw_spin_unlock+0x12/0x30 [ 8740.083592] do_sys_openat2+0x2b4/0x410 [ 8740.083594] do_sys_open+0x73/0xa0 [ 8740.083597] __x64_sys_openat+0x24/0x30 [ 8740.083599] do_syscall_64+0xc1/0x440 [ 8740.083604] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8740.083608] RIP: 0033:0x7fdc0afbf332 [ 8740.083615] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8740.083618] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8740.083622] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8740.083623] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8740.083625] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8740.083627] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8740.083628] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8740.738971] ? tick_sched_handle.isra.18+0x27/0x90 [ 8740.745717] ? tick_sched_timer+0x53/0xb0 [ 8740.748943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8740.753381] ? tick_sched_handle.isra.18+0x90/0x90 [ 8740.767954] ? hrtimer_interrupt+0x128/0x2c0 [ 8740.770775] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8740.773712] ? apic_timer_interrupt+0xf/0x20 [ 8740.780400] [ 8740.781550] ? native_safe_halt+0x2e/0x50 [ 8740.786541] ? kvm_wait+0xcd/0xe0 [ 8740.792494] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8740.798268] ? dump_stack+0xf6/0x10e [ 8740.803012] ? do_raw_spin_lock+0x177/0x190 [ 8740.808586] ? _raw_spin_lock+0x21/0x30 [ 8740.811170] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8740.817179] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8740.819181] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8740.821445] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8740.838036] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8740.847669] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8740.849871] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8740.853001] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8740.854471] ? do_wait_intr+0xf0/0xf0 [ 8740.855778] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8740.857768] ? kthread+0x1d1/0x200 [ 8740.864653] ? set_kthread_struct+0x70/0x70 [ 8740.866363] ? ret_from_fork+0x1f/0x30 [ 8752.054733] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8752.062773] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8752.105310] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8752.111163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8752.116497] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8752.119840] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8752.122577] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8752.130894] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8752.134603] RAX: 0000000000006488 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8752.137840] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8752.141792] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8752.144845] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8752.148367] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8752.151062] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8752.154792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8752.157555] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8752.161116] Call Trace: [ 8752.162565] [ 8752.163596] ? show_regs.cold.9+0x22/0x2f [ 8752.165569] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8752.167793] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8752.169857] ? watchdog+0x40/0x40 [ 8752.171643] ? hrtimer_interrupt+0x128/0x2c0 [ 8752.173747] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8752.175817] ? apic_timer_interrupt+0xf/0x20 [ 8752.177511] [ 8752.178444] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8752.182436] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8752.186164] do_raw_spin_lock+0x177/0x190 [ 8752.188789] _raw_spin_lock+0x21/0x30 [ 8752.192311] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8752.197332] process_one_work+0x2c8/0x700 [ 8752.199594] worker_thread+0x296/0x6e0 [ 8752.202510] ? rescuer_thread+0x570/0x570 [ 8752.204321] kthread+0x1d1/0x200 [ 8752.208024] ? set_kthread_struct+0x70/0x70 [ 8752.211321] ret_from_fork+0x1f/0x30 [ 8764.048026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8764.054740] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 8764.054969] Modules linked in: [ 8764.059615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [ 8764.062262] lustre(O) osp(O) [ 8764.094631] nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8764.094677] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8764.094685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8764.094688] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8764.094701] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8764.094706] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8764.094711] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8764.094714] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8764.094716] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8764.094719] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8764.094721] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8764.094725] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8764.094728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8764.094731] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8764.094739] Call Trace: [ 8764.094744] [ 8764.094749] ? show_regs.cold.9+0x22/0x2f [ 8764.094759] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8764.094766] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8764.094770] ? watchdog+0x40/0x40 [ 8764.094775] ? hrtimer_interrupt+0x128/0x2c0 [ 8764.094780] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8764.094787] ? apic_timer_interrupt+0xf/0x20 [ 8764.094792] [ 8764.094795] ? native_safe_halt+0x2e/0x50 [ 8764.094799] kvm_wait+0xcd/0xe0 [ 8764.094807] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8764.094816] do_raw_spin_lock+0x177/0x190 [ 8764.094820] _raw_spin_lock+0x21/0x30 [ 8764.094825] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8764.097492] ? do_raw_spin_unlock+0x75/0x190 [ 8764.097503] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8764.098361] ? do_wait_intr+0xf0/0xf0 [ 8764.098371] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8764.098589] mdc_close+0x53b/0xd70 [mdc] [ 8764.098631] lmv_close+0x18d/0x490 [lmv] [ 8764.100187] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8764.100301] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8764.100385] ll_md_close+0x274/0x800 [lustre] [ 8764.100486] ll_file_release+0x128/0xc50 [lustre] [ 8764.100577] __fput+0x10d/0x440 [ 8764.100586] ____fput+0x12/0x20 [ 8764.100591] task_work_run+0xc6/0x110 [ 8764.100600] exit_to_usermode_loop+0x1dd/0x1f0 [ 8764.100608] do_syscall_64+0x3d6/0x440 [ 8764.100613] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8764.100621] RIP: 0033:0x7fdc0afbfcd8 [ 8764.100629] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8764.100633] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8764.100639] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8764.100642] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8764.100644] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8764.100646] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8764.100649] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8764.585676] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8764.633431] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8764.642290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8764.650366] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8764.652037] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8764.696718] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8764.708375] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8764.720444] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8764.733455] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8764.735782] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8764.746854] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8764.760854] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8764.778449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8764.787214] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8764.800737] Call Trace: [ 8764.803502] [ 8764.807110] ? show_regs.cold.9+0x22/0x2f [ 8764.814709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8764.822216] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8764.825990] ? watchdog+0x40/0x40 [ 8764.832975] ? hrtimer_interrupt+0x128/0x2c0 [ 8764.840385] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8764.847091] ? apic_timer_interrupt+0xf/0x20 [ 8764.849156] [ 8764.851199] ? native_safe_halt+0x2e/0x50 [ 8764.854251] kvm_wait+0xcd/0xe0 [ 8764.855587] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8764.860555] ? dump_stack+0xf6/0x10e [ 8764.864643] do_raw_spin_lock+0x177/0x190 [ 8764.866825] _raw_spin_lock+0x21/0x30 [ 8764.868203] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8764.874385] obd_mod_free+0x46/0x140 [ptlrpc] [ 8764.885783] mdc_commit_open+0x91/0xb0 [mdc] [ 8764.889281] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8764.894878] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8764.897081] after_reply+0xe2c/0x1780 [ptlrpc] [ 8764.904382] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8764.914871] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8764.921682] ? do_wait_intr+0xf0/0xf0 [ 8764.930585] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8764.939644] kthread+0x1d1/0x200 [ 8764.940941] ? set_kthread_struct+0x70/0x70 [ 8764.942830] ret_from_fork+0x1f/0x30 [ 8764.944768] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6740s! [ 8764.949844] Showing busy workqueues and worker pools: [ 8764.956185] workqueue events: flags=0x0 [ 8764.959659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8764.963159] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8764.966434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8764.971941] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8764.977521] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8764.983151] pending: kfree_rcu_monitor [ 8764.987792] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8764.997148] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8765.012904] workqueue events_long: flags=0x0 [ 8765.016434] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8765.024752] pending: lu_objects_flush [obdclass] [ 8765.035593] workqueue events_unbound: flags=0x2 [ 8765.041938] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8765.057110] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8765.076563] workqueue events_power_efficient: flags=0x80 [ 8765.080327] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8765.091612] pending: neigh_periodic_work, check_lifetime [ 8765.103248] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8765.106714] pending: do_cache_clean, neigh_periodic_work [ 8765.109571] workqueue mm_percpu_wq: flags=0x8 [ 8765.121773] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8765.131790] pending: lru_add_drain_per_cpu [ 8765.137560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8765.142713] pending: vmstat_update, lru_add_drain_per_cpu [ 8765.150047] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8765.155943] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8765.163968] workqueue writeback: flags=0x4a [ 8765.168144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8765.176334] pending: wb_workfn, wb_workfn [ 8765.181953] workqueue xprtiod: flags=0xa [ 8765.185773] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8765.195842] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8765.203813] workqueue ipv6_addrconf: flags=0x40008 [ 8765.206716] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8765.218513] pending: addrconf_verify_work [ 8765.224729] workqueue knbd0-recv: flags=0x1a [ 8765.229516] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8765.236118] in-flight: 655:recv_work [ 8765.242147] workqueue ptlrpc_pinger: flags=0x2 [ 8765.246626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8765.250908] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8765.257420] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6740s workers=4 idle: 36396 38 11188 [ 8765.261497] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8755s workers=2 idle: 658 [ 8768.052073] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8768.054769] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8768.078951] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8768.086695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8768.094249] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8768.097700] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8768.117204] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8768.123467] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8768.132663] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8768.141662] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8768.148995] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8768.160687] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8768.168905] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8768.172524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8768.176969] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8768.184938] Call Trace: [ 8768.185551] [ 8768.186035] ? show_regs.cold.9+0x22/0x2f [ 8768.188279] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8768.190577] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8768.196445] ? watchdog+0x40/0x40 [ 8768.199610] ? hrtimer_interrupt+0x128/0x2c0 [ 8768.204531] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8768.207913] ? apic_timer_interrupt+0xf/0x20 [ 8768.209847] [ 8768.210824] ? native_safe_halt+0x2e/0x50 [ 8768.213628] kvm_wait+0xcd/0xe0 [ 8768.217581] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8768.220799] do_raw_spin_lock+0x177/0x190 [ 8768.224502] _raw_spin_lock+0x21/0x30 [ 8768.226423] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8768.231436] ? do_raw_spin_unlock+0x75/0x190 [ 8768.236547] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8768.241434] ? do_wait_intr+0xf0/0xf0 [ 8768.243360] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8768.245229] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8768.250544] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8768.256272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8768.260692] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8768.268579] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8768.270105] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8768.274628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8768.276059] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8768.282619] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8768.286892] lmv_intent_open+0x338/0x1150 [lmv] [ 8768.291867] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8768.297803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8768.299621] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8768.307981] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8768.309682] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8768.316344] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8768.318029] ? do_raw_spin_unlock+0x75/0x190 [ 8768.330042] ? _raw_spin_unlock+0x12/0x30 [ 8768.332562] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8768.335515] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8768.337477] ll_atomic_open+0x36a/0xd70 [lustre] [ 8768.340190] lookup_open+0x356/0xb40 [ 8768.341641] do_last+0x313/0xfc0 [ 8768.346887] ? nd_jump_root+0xe5/0x160 [ 8768.351098] ? path_init+0x437/0x520 [ 8768.352512] path_openat+0xf7/0x500 [ 8768.354904] ? do_raw_spin_unlock+0x75/0x190 [ 8768.358104] do_filp_open+0x99/0x140 [ 8768.360425] ? getname_flags+0x6e/0x330 [ 8768.362792] ? __check_object_size+0xff/0x256 [ 8768.365744] ? do_raw_spin_unlock+0x75/0x190 [ 8768.371782] ? _raw_spin_unlock+0x12/0x30 [ 8768.379032] do_sys_openat2+0x2b4/0x410 [ 8768.380228] do_sys_open+0x73/0xa0 [ 8768.381303] __x64_sys_openat+0x24/0x30 [ 8768.386911] do_syscall_64+0xc1/0x440 [ 8768.390515] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8768.400212] RIP: 0033:0x7fdc0afbf332 [ 8768.401445] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8768.416662] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8768.419799] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8768.426270] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8768.430130] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8768.436169] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8768.440148] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8780.051143] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 8780.055440] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8780.081223] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8780.087240] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8780.092431] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8780.096873] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8780.100717] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8780.111234] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8780.115920] RAX: 0000000000006708 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8780.119319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8780.125899] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8780.135983] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8780.143966] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8780.147651] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8780.152873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8780.154353] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8780.159223] Call Trace: [ 8780.164947] [ 8780.168682] ? show_regs.cold.9+0x22/0x2f [ 8780.172465] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8780.174852] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8780.180032] ? watchdog+0x40/0x40 [ 8780.181160] ? hrtimer_interrupt+0x128/0x2c0 [ 8780.187038] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8780.192646] ? apic_timer_interrupt+0xf/0x20 [ 8780.200302] [ 8780.202482] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8780.211760] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8780.215790] do_raw_spin_lock+0x177/0x190 [ 8780.218909] _raw_spin_lock+0x21/0x30 [ 8780.221989] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8780.225076] process_one_work+0x2c8/0x700 [ 8780.227729] worker_thread+0x296/0x6e0 [ 8780.231766] ? rescuer_thread+0x570/0x570 [ 8780.235282] kthread+0x1d1/0x200 [ 8780.236231] ? set_kthread_struct+0x70/0x70 [ 8780.242726] ret_from_fork+0x1f/0x30 [ 8792.050542] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 8792.054021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 8792.063037] Modules linked in: [ 8792.068388] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8792.068606] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8792.068614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8792.068620] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8792.068634] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8792.068639] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8792.068643] RAX: 00000000000014a7 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8792.068645] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8792.068647] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8792.068649] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8792.068651] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8792.068654] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8792.068657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8792.068659] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8792.068668] Call Trace: [ 8792.068673] [ 8792.068678] ? show_regs.cold.9+0x22/0x2f [ 8792.068687] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8792.068695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8792.068699] ? watchdog+0x40/0x40 [ 8792.068704] ? hrtimer_interrupt+0x128/0x2c0 [ 8792.072585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8792.072616] ? apic_timer_interrupt+0xf/0x20 [ 8792.072621] [ 8792.072625] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8792.072637] do_raw_spin_lock+0x177/0x190 [ 8792.072643] _raw_spin_lock+0x21/0x30 [ 8792.072648] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8792.073712] ? do_raw_spin_unlock+0x75/0x190 [ 8792.073728] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8792.074006] ? do_wait_intr+0xf0/0xf0 [ 8792.074015] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8792.074506] mdc_close+0x53b/0xd70 [mdc] [ 8792.074736] lmv_close+0x18d/0x490 [lmv] [ 8792.074767] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8792.074880] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8792.074977] ll_md_close+0x274/0x800 [lustre] [ 8792.075075] ll_file_release+0x128/0xc50 [lustre] [ 8792.077097] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8792.077168] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8792.077174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8792.077178] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8792.077587] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8792.077591] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 [ 8792.154172] __fput+0x10d/0x440 [ 8792.154192] ____fput+0x12/0x20 [ 8792.154198] task_work_run+0xc6/0x110 [ 8792.154205] exit_to_usermode_loop+0x1dd/0x1f0 [ 8792.154212] do_syscall_64+0x3d6/0x440 [ 8792.154217] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8792.154225] RIP: 0033:0x7fdc0afbfcd8 [ 8792.154247] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8792.154251] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8792.154257] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8792.154264] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8792.154267] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8792.154270] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8792.154272] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8792.684945] ORIG_RAX: ffffffffffffff13 [ 8792.688628] RAX: 0000000000005984 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8792.694528] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8792.702253] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8792.709698] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8792.718464] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8792.727781] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8792.735433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8792.737093] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8792.745829] Call Trace: [ 8792.746705] [ 8792.750121] ? show_regs.cold.9+0x22/0x2f [ 8792.754240] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8792.757380] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8792.760199] ? watchdog+0x40/0x40 [ 8792.762008] ? hrtimer_interrupt+0x128/0x2c0 [ 8792.763494] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8792.765300] ? apic_timer_interrupt+0xf/0x20 [ 8792.766974] [ 8792.767769] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8792.769871] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8792.775437] ? dump_stack+0xf6/0x10e [ 8792.781231] do_raw_spin_lock+0x177/0x190 [ 8792.783002] _raw_spin_lock+0x21/0x30 [ 8792.788492] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8792.794269] obd_mod_free+0x46/0x140 [ptlrpc] [ 8792.798040] mdc_commit_open+0x91/0xb0 [mdc] [ 8792.800370] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8792.803189] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8792.807139] after_reply+0xe2c/0x1780 [ptlrpc] [ 8792.809843] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8792.812809] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8792.814810] ? do_wait_intr+0xf0/0xf0 [ 8792.816704] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8792.819057] kthread+0x1d1/0x200 [ 8792.825467] ? set_kthread_struct+0x70/0x70 [ 8792.830164] ret_from_fork+0x1f/0x30 [ 8795.615026] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6770s! [ 8795.618163] Showing busy workqueues and worker pools: [ 8795.620686] workqueue events: flags=0x0 [ 8795.622510] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8795.625182] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8795.628094] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8795.638559] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8795.646151] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8795.658904] pending: kfree_rcu_monitor [ 8795.663605] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8795.666739] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8795.680788] workqueue events_long: flags=0x0 [ 8795.681993] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8795.688909] pending: lu_objects_flush [obdclass] [ 8795.692753] workqueue events_unbound: flags=0x2 [ 8795.697400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8795.702180] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8795.718228] workqueue events_power_efficient: flags=0x80 [ 8795.726287] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8795.733702] pending: neigh_periodic_work, check_lifetime [ 8795.736270] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8795.744109] pending: do_cache_clean, neigh_periodic_work [ 8795.754628] workqueue mm_percpu_wq: flags=0x8 [ 8795.756073] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8795.769505] pending: lru_add_drain_per_cpu [ 8795.781794] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8795.795117] pending: vmstat_update, lru_add_drain_per_cpu [ 8795.803808] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8795.808590] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8795.813888] workqueue writeback: flags=0x4a [ 8795.816359] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8795.821365] pending: wb_workfn, wb_workfn [ 8795.824266] workqueue xprtiod: flags=0xa [ 8795.831589] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8795.840096] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8795.845772] workqueue ipv6_addrconf: flags=0x40008 [ 8795.848826] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8795.854857] pending: addrconf_verify_work [ 8795.859057] workqueue knbd0-recv: flags=0x1a [ 8795.861433] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8795.865186] in-flight: 655:recv_work [ 8795.867764] workqueue ptlrpc_pinger: flags=0x2 [ 8795.870446] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8795.877156] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8795.881031] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6770s workers=4 idle: 36396 38 11188 [ 8795.887102] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8786s workers=2 idle: 658 [ 8796.052037] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8796.054843] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8796.110062] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8796.127518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8796.139886] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8796.147534] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8796.166235] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8796.177177] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8796.185055] RDX: 00000000000000a9 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8796.187506] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8796.199038] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8796.204558] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8796.208252] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8796.214754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8796.216937] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8796.219562] Call Trace: [ 8796.220553] [ 8796.221264] ? show_regs.cold.9+0x22/0x2f [ 8796.222941] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8796.224798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8796.226679] ? watchdog+0x40/0x40 [ 8796.227926] ? hrtimer_interrupt+0x128/0x2c0 [ 8796.229571] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8796.231568] ? apic_timer_interrupt+0xf/0x20 [ 8796.233196] [ 8796.234126] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8796.236044] do_raw_spin_lock+0x177/0x190 [ 8796.237413] _raw_spin_lock+0x21/0x30 [ 8796.238974] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8796.240950] ? do_raw_spin_unlock+0x75/0x190 [ 8796.242637] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8796.244682] ? do_wait_intr+0xf0/0xf0 [ 8796.246073] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8796.248183] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8796.252645] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8796.254328] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8796.256241] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8796.258198] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8796.259877] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8796.261514] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8796.263425] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8796.265763] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8796.267691] lmv_intent_open+0x338/0x1150 [lmv] [ 8796.269323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8796.271310] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8796.273302] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8796.274996] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8796.276888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8796.279037] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8796.280841] ? do_raw_spin_unlock+0x75/0x190 [ 8796.282434] ? _raw_spin_unlock+0x12/0x30 [ 8796.283932] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8796.285878] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8796.287773] ll_atomic_open+0x36a/0xd70 [lustre] [ 8796.289395] lookup_open+0x356/0xb40 [ 8796.290681] do_last+0x313/0xfc0 [ 8796.291921] ? nd_jump_root+0xe5/0x160 [ 8796.293254] ? path_init+0x437/0x520 [ 8796.294626] path_openat+0xf7/0x500 [ 8796.295777] ? do_raw_spin_unlock+0x75/0x190 [ 8796.297372] do_filp_open+0x99/0x140 [ 8796.298815] ? getname_flags+0x6e/0x330 [ 8796.300186] ? __check_object_size+0xff/0x256 [ 8796.301751] ? do_raw_spin_unlock+0x75/0x190 [ 8796.303259] ? _raw_spin_unlock+0x12/0x30 [ 8796.304760] do_sys_openat2+0x2b4/0x410 [ 8796.306070] do_sys_open+0x73/0xa0 [ 8796.307361] __x64_sys_openat+0x24/0x30 [ 8796.308917] do_syscall_64+0xc1/0x440 [ 8796.310086] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8796.311365] RIP: 0033:0x7fdc0afbf332 [ 8796.312304] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8796.321082] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8796.327875] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8796.330294] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8796.332851] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8796.336855] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8796.342663] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8808.051079] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 8808.057331] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8808.110209] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8808.120990] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8808.131713] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8808.140914] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8808.143560] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8808.168383] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8808.175195] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8808.183789] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8808.187427] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8808.198838] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8808.208913] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8808.222559] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8808.230155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8808.236664] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8808.242019] Call Trace: [ 8808.243760] [ 8808.247571] ? show_regs.cold.9+0x22/0x2f [ 8808.249622] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8808.256842] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8808.261930] ? watchdog+0x40/0x40 [ 8808.264248] ? hrtimer_interrupt+0x128/0x2c0 [ 8808.266641] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8808.269343] ? apic_timer_interrupt+0xf/0x20 [ 8808.272375] [ 8808.279898] ? native_safe_halt+0x2e/0x50 [ 8808.284397] kvm_wait+0xcd/0xe0 [ 8808.287144] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8808.290343] do_raw_spin_lock+0x177/0x190 [ 8808.292865] _raw_spin_lock+0x21/0x30 [ 8808.294716] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8808.299126] process_one_work+0x2c8/0x700 [ 8808.310325] worker_thread+0x296/0x6e0 [ 8808.319900] ? rescuer_thread+0x570/0x570 [ 8808.322760] kthread+0x1d1/0x200 [ 8808.323933] ? set_kthread_struct+0x70/0x70 [ 8808.332748] ret_from_fork+0x1f/0x30 [ 8820.048062] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 8820.051548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8820.056272] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 8820.095894] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8820.095909] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8820.095912] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8820.095928] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8820.095932] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8820.095937] RAX: 0000000000005413 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8820.095940] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8820.095942] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8820.095945] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8820.095947] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8820.095951] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8820.095954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8820.095957] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8820.095965] Call Trace: [ 8820.095971] [ 8820.095976] ? show_regs.cold.9+0x22/0x2f [ 8820.095987] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8820.095993] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8820.095997] ? watchdog+0x40/0x40 [ 8820.096002] ? hrtimer_interrupt+0x128/0x2c0 [ 8820.096007] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8820.096015] ? apic_timer_interrupt+0xf/0x20 [ 8820.096021] [ 8820.096024] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8820.096030] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8820.096033] ? dump_stack+0xf6/0x10e [ 8820.096039] do_raw_spin_lock+0x177/0x190 [ 8820.096043] _raw_spin_lock+0x21/0x30 [ 8820.096052] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8820.096320] obd_mod_free+0x46/0x140 [ptlrpc] [ 8820.103313] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8820.103408] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8820.103494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8820.103497] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8820.103509] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8820.103513] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8820.103522] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8820.103524] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8820.103527] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8820.103529] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8820.103532] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8820.103535] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8820.103538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8820.103541] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8820.103549] Call Trace: [ 8820.103554] [ 8820.103558] ? show_regs.cold.9+0x22/0x2f [ 8820.103573] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8820.103580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8820.103585] ? watchdog+0x40/0x40 [ 8820.103589] ? hrtimer_interrupt+0x128/0x2c0 [ 8820.103593] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8820.103599] ? apic_timer_interrupt+0xf/0x20 [ 8820.103604] [ 8820.103607] ? native_safe_halt+0x2e/0x50 [ 8820.103612] kvm_wait+0xcd/0xe0 [ 8820.103619] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8820.103628] do_raw_spin_lock+0x177/0x190 [ 8820.103632] _raw_spin_lock+0x21/0x30 [ 8820.103637] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8820.104922] ? do_raw_spin_unlock+0x75/0x190 [ 8820.104935] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8820.105490] ? do_wait_intr+0xf0/0xf0 [ 8820.105499] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8820.121181] mdc_commit_open+0x91/0xb0 [mdc] [ 8820.121231] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8820.122271] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8820.133331] mdc_close+0x53b/0xd70 [mdc] [ 8820.133391] lmv_close+0x18d/0x490 [lmv] [ 8820.133509] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8820.133663] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8820.133764] ll_md_close+0x274/0x800 [lustre] [ 8820.133867] ll_file_release+0x128/0xc50 [lustre] [ 8820.135114] __fput+0x10d/0x440 [ 8820.135124] ____fput+0x12/0x20 [ 8820.135129] task_work_run+0xc6/0x110 [ 8820.135138] exit_to_usermode_loop+0x1dd/0x1f0 [ 8820.135145] do_syscall_64+0x3d6/0x440 [ 8820.135149] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8820.135283] RIP: 0033:0x7fdc0afbfcd8 [ 8820.135290] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8820.135294] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8820.135300] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8820.135303] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8820.135321] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8820.135324] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8820.135327] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8820.672683] after_reply+0xe2c/0x1780 [ptlrpc] [ 8820.681796] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8820.692115] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8820.695527] ? do_wait_intr+0xf0/0xf0 [ 8820.699520] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8820.708569] kthread+0x1d1/0x200 [ 8820.711814] ? set_kthread_struct+0x70/0x70 [ 8820.721319] ret_from_fork+0x1f/0x30 [ 8824.052629] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 8824.055902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8824.074750] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8824.082597] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8824.089174] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8824.094461] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8824.106149] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8824.113690] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8824.116513] RDX: 00000000000031b2 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8824.124766] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8824.129213] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8824.134792] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8824.139337] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8824.145454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8824.148415] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8824.151286] Call Trace: [ 8824.153616] [ 8824.154510] ? show_regs.cold.9+0x22/0x2f [ 8824.156476] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8824.158752] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8824.160933] ? watchdog+0x40/0x40 [ 8824.162611] ? hrtimer_interrupt+0x128/0x2c0 [ 8824.164446] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8824.167387] ? apic_timer_interrupt+0xf/0x20 [ 8824.169660] [ 8824.170621] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8824.172840] do_raw_spin_lock+0x177/0x190 [ 8824.174455] _raw_spin_lock+0x21/0x30 [ 8824.175756] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8824.182957] ? do_raw_spin_unlock+0x75/0x190 [ 8824.184242] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8824.191204] ? do_wait_intr+0xf0/0xf0 [ 8824.193331] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8824.199282] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8824.204379] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8824.207750] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8824.209725] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8824.214246] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8824.216047] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8824.218381] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8824.224278] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8824.227143] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8824.233281] lmv_intent_open+0x338/0x1150 [lmv] [ 8824.236031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8824.238542] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8824.241467] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8824.243197] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8824.244679] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8824.247014] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8824.248443] ? do_raw_spin_unlock+0x75/0x190 [ 8824.249612] ? _raw_spin_unlock+0x12/0x30 [ 8824.251232] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8824.253309] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8824.255413] ll_atomic_open+0x36a/0xd70 [lustre] [ 8824.257455] lookup_open+0x356/0xb40 [ 8824.259098] do_last+0x313/0xfc0 [ 8824.260399] ? nd_jump_root+0xe5/0x160 [ 8824.261941] ? path_init+0x437/0x520 [ 8824.263388] path_openat+0xf7/0x500 [ 8824.264655] ? do_raw_spin_unlock+0x75/0x190 [ 8824.266580] do_filp_open+0x99/0x140 [ 8824.268447] ? getname_flags+0x6e/0x330 [ 8824.270197] ? __check_object_size+0xff/0x256 [ 8824.272001] ? do_raw_spin_unlock+0x75/0x190 [ 8824.273888] ? _raw_spin_unlock+0x12/0x30 [ 8824.275447] do_sys_openat2+0x2b4/0x410 [ 8824.277122] do_sys_open+0x73/0xa0 [ 8824.278579] __x64_sys_openat+0x24/0x30 [ 8824.280274] do_syscall_64+0xc1/0x440 [ 8824.281623] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8824.283839] RIP: 0033:0x7fdc0afbf332 [ 8824.285642] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8824.293152] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8824.296288] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8824.300337] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8824.303712] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8824.306558] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8824.309300] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8826.335075] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6801s! [ 8826.338388] Showing busy workqueues and worker pools: [ 8826.340135] workqueue events: flags=0x0 [ 8826.347873] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8826.356804] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8826.365633] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8826.370868] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8826.385243] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8826.403331] pending: kfree_rcu_monitor [ 8826.406965] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8826.414966] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8826.430173] workqueue events_long: flags=0x0 [ 8826.436494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8826.446843] pending: lu_objects_flush [obdclass] [ 8826.454074] workqueue events_unbound: flags=0x2 [ 8826.459026] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8826.465817] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8826.475595] workqueue events_power_efficient: flags=0x80 [ 8826.479903] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8826.484869] pending: neigh_periodic_work, check_lifetime [ 8826.490319] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8826.498045] pending: do_cache_clean, neigh_periodic_work [ 8826.500267] workqueue mm_percpu_wq: flags=0x8 [ 8826.511265] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8826.520739] pending: lru_add_drain_per_cpu [ 8826.523726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8826.528135] pending: vmstat_update, lru_add_drain_per_cpu [ 8826.534910] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8826.541374] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8826.544394] workqueue writeback: flags=0x4a [ 8826.547634] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8826.553406] pending: wb_workfn, wb_workfn [ 8826.558969] workqueue xprtiod: flags=0xa [ 8826.562829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8826.568197] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8826.572827] workqueue ipv6_addrconf: flags=0x40008 [ 8826.574750] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8826.579426] pending: addrconf_verify_work [ 8826.583222] workqueue knbd0-recv: flags=0x1a [ 8826.585257] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8826.588011] in-flight: 655:recv_work [ 8826.592427] workqueue ptlrpc_pinger: flags=0x2 [ 8826.594600] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8826.597601] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8826.604259] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6801s workers=4 idle: 36396 38 11188 [ 8826.616888] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8817s workers=2 idle: 658 [ 8836.051061] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 8836.055013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8836.076014] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8836.080461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8836.084455] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8836.087120] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8836.092291] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8836.105537] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8836.111707] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8836.114945] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8836.119682] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8836.128542] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8836.132677] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8836.136354] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8836.140291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8836.145188] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8836.149642] Call Trace: [ 8836.150645] [ 8836.151762] ? show_regs.cold.9+0x22/0x2f [ 8836.153691] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8836.156077] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8836.158963] ? watchdog+0x40/0x40 [ 8836.160611] ? hrtimer_interrupt+0x128/0x2c0 [ 8836.162577] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8836.165518] ? apic_timer_interrupt+0xf/0x20 [ 8836.167779] [ 8836.168743] ? native_safe_halt+0x2e/0x50 [ 8836.170564] kvm_wait+0xcd/0xe0 [ 8836.172478] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8836.176048] do_raw_spin_lock+0x177/0x190 [ 8836.180177] _raw_spin_lock+0x21/0x30 [ 8836.182883] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8836.187199] process_one_work+0x2c8/0x700 [ 8836.189351] worker_thread+0x296/0x6e0 [ 8836.191195] ? rescuer_thread+0x570/0x570 [ 8836.194150] kthread+0x1d1/0x200 [ 8836.195444] ? set_kthread_struct+0x70/0x70 [ 8836.199527] ret_from_fork+0x1f/0x30 [ 8848.062664] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8848.072089] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8848.109451] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8848.120337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8848.129263] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8848.135690] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8848.152074] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8848.156565] RAX: 000000000000664a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8848.162039] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8848.168794] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8848.172481] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8848.178091] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8848.181415] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8848.184993] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8848.188080] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8848.193364] Call Trace: [ 8848.194252] [ 8848.195250] ? show_regs.cold.9+0x22/0x2f [ 8848.197453] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8848.199895] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8848.206201] ? watchdog+0x40/0x40 [ 8848.209356] ? hrtimer_interrupt+0x128/0x2c0 [ 8848.211892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8848.213885] ? apic_timer_interrupt+0xf/0x20 [ 8848.215977] [ 8848.217194] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8848.219411] do_raw_spin_lock+0x177/0x190 [ 8848.221324] _raw_spin_lock+0x21/0x30 [ 8848.222832] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8848.225436] ? do_raw_spin_unlock+0x75/0x190 [ 8848.227209] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8848.229454] ? do_wait_intr+0xf0/0xf0 [ 8848.231306] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8848.233535] mdc_close+0x53b/0xd70 [mdc] [ 8848.235225] lmv_close+0x18d/0x490 [lmv] [ 8848.237097] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8848.239989] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8848.242282] ll_md_close+0x274/0x800 [lustre] [ 8848.244212] ll_file_release+0x128/0xc50 [lustre] [ 8848.246087] __fput+0x10d/0x440 [ 8848.247387] ____fput+0x12/0x20 [ 8848.248981] task_work_run+0xc6/0x110 [ 8848.250400] exit_to_usermode_loop+0x1dd/0x1f0 [ 8848.252530] do_syscall_64+0x3d6/0x440 [ 8848.254251] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8848.256862] RIP: 0033:0x7fdc0afbfcd8 [ 8848.258422] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8848.267301] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8848.271127] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8848.275005] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8848.278542] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8848.282244] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8848.285397] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8852.048029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 8852.050743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8852.052220] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 8852.070628] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8852.070642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8852.070645] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8852.070658] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8852.070663] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8852.070668] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8852.070670] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8852.070672] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8852.070674] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8852.070677] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8852.070680] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8852.070684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8852.070686] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8852.070694] Call Trace: [ 8852.070700] [ 8852.070705] ? show_regs.cold.9+0x22/0x2f [ 8852.070715] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8852.070721] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8852.074068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8852.074164] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8852.074174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8852.074176] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8852.074192] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8852.074197] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8852.074202] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8852.074205] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8852.074207] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8852.074209] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8852.074216] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8852.074223] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8852.074227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8852.074230] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8852.074237] Call Trace: [ 8852.074242] [ 8852.074251] ? show_regs.cold.9+0x22/0x2f [ 8852.074261] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8852.074267] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8852.074271] ? watchdog+0x40/0x40 [ 8852.074275] ? hrtimer_interrupt+0x128/0x2c0 [ 8852.074278] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8852.074285] ? apic_timer_interrupt+0xf/0x20 [ 8852.074289] [ 8852.074292] ? native_safe_halt+0x2e/0x50 [ 8852.074296] kvm_wait+0xcd/0xe0 [ 8852.074303] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8852.074310] do_raw_spin_lock+0x177/0x190 [ 8852.074314] _raw_spin_lock+0x21/0x30 [ 8852.074320] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8852.074640] ? do_raw_spin_unlock+0x75/0x190 [ 8852.074650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8852.074984] ? do_wait_intr+0xf0/0xf0 [ 8852.074993] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8852.075268] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8852.075563] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8852.075830] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8852.082121] ? watchdog+0x40/0x40 [ 8852.082142] ? hrtimer_interrupt+0x128/0x2c0 [ 8852.082151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8852.082160] ? apic_timer_interrupt+0xf/0x20 [ 8852.082164] [ 8852.082169] ? native_safe_halt+0x2e/0x50 [ 8852.082173] kvm_wait+0xcd/0xe0 [ 8852.082181] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8852.082193] ? dump_stack+0xf6/0x10e [ 8852.082198] do_raw_spin_lock+0x177/0x190 [ 8852.082202] _raw_spin_lock+0x21/0x30 [ 8852.082212] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8852.082497] obd_mod_free+0x46/0x140 [ptlrpc] [ 8852.082700] mdc_commit_open+0x91/0xb0 [mdc] [ 8852.082735] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8852.082972] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8852.083217] after_reply+0xe2c/0x1780 [ptlrpc] [ 8852.083437] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8852.083661] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8852.088316] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8852.088365] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8852.088393] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8852.088422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8852.088592] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8852.088862] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8852.088902] lmv_intent_open+0x338/0x1150 [lmv] [ 8852.088939] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8852.089049] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8852.092141] ? do_wait_intr+0xf0/0xf0 [ 8852.092170] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8852.106942] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8852.106987] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8852.107168] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8852.107257] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8852.107399] ? do_raw_spin_unlock+0x75/0x190 [ 8852.107408] ? _raw_spin_unlock+0x12/0x30 [ 8852.107415] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8852.107445] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8852.107467] ll_atomic_open+0x36a/0xd70 [lustre] [ 8852.107610] lookup_open+0x356/0xb40 [ 8852.107620] do_last+0x313/0xfc0 [ 8852.107625] ? nd_jump_root+0xe5/0x160 [ 8852.107630] ? path_init+0x437/0x520 [ 8852.107635] path_openat+0xf7/0x500 [ 8852.107640] ? do_raw_spin_unlock+0x75/0x190 [ 8852.107644] do_filp_open+0x99/0x140 [ 8852.107650] ? getname_flags+0x6e/0x330 [ 8852.107655] ? __check_object_size+0xff/0x256 [ 8852.107663] ? do_raw_spin_unlock+0x75/0x190 [ 8852.107667] ? _raw_spin_unlock+0x12/0x30 [ 8852.107672] do_sys_openat2+0x2b4/0x410 [ 8852.107676] do_sys_open+0x73/0xa0 [ 8852.107681] __x64_sys_openat+0x24/0x30 [ 8852.107685] do_syscall_64+0xc1/0x440 [ 8852.107692] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8852.107699] RIP: 0033:0x7fdc0afbf332 [ 8852.107706] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8852.107710] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8852.107716] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8852.107718] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8852.107720] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8852.107722] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8852.107725] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8852.582122] kthread+0x1d1/0x200 [ 8852.584785] ? set_kthread_struct+0x70/0x70 [ 8852.586836] ret_from_fork+0x1f/0x30 [ 8857.055535] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6832s! [ 8857.060257] Showing busy workqueues and worker pools: [ 8857.061510] workqueue events: flags=0x0 [ 8857.063705] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8857.065019] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8857.066554] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8857.068549] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8857.071249] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8857.072826] pending: kfree_rcu_monitor [ 8857.073660] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8857.074769] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8857.078688] workqueue events_long: flags=0x0 [ 8857.079832] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8857.081637] pending: lu_objects_flush [obdclass] [ 8857.083721] workqueue events_unbound: flags=0x2 [ 8857.085097] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8857.087573] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8857.091244] workqueue events_power_efficient: flags=0x80 [ 8857.092631] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8857.094566] pending: neigh_periodic_work, check_lifetime [ 8857.096183] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8857.098234] pending: do_cache_clean, neigh_periodic_work [ 8857.099859] workqueue mm_percpu_wq: flags=0x8 [ 8857.101192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8857.102952] pending: lru_add_drain_per_cpu [ 8857.104293] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8857.105980] pending: vmstat_update, lru_add_drain_per_cpu [ 8857.107721] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8857.110505] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8857.112924] workqueue writeback: flags=0x4a [ 8857.114659] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8857.116663] pending: wb_workfn, wb_workfn [ 8857.118480] workqueue xprtiod: flags=0xa [ 8857.119873] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8857.122238] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8857.125073] workqueue ipv6_addrconf: flags=0x40008 [ 8857.127104] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8857.129722] pending: addrconf_verify_work [ 8857.131579] workqueue knbd0-recv: flags=0x1a [ 8857.133275] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8857.135864] in-flight: 655:recv_work [ 8857.137566] workqueue ptlrpc_pinger: flags=0x2 [ 8857.139478] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8857.141828] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8857.144378] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6832s workers=4 idle: 36396 38 11188 [ 8857.147692] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8847s workers=2 idle: 658 [ 8864.051888] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 8864.061355] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8864.117478] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8864.125762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8864.133244] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8864.140158] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8864.149049] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8864.182244] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8864.189504] RAX: 0000000000000006 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8864.197755] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8864.205613] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8864.213888] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8864.217963] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8864.222140] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8864.226422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8864.229754] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8864.234358] Call Trace: [ 8864.235598] [ 8864.236224] ? show_regs.cold.9+0x22/0x2f [ 8864.238606] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8864.241258] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8864.246223] ? watchdog+0x40/0x40 [ 8864.247958] ? hrtimer_interrupt+0x128/0x2c0 [ 8864.249626] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8864.251142] ? apic_timer_interrupt+0xf/0x20 [ 8864.252497] [ 8864.253364] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8864.255629] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8864.257335] do_raw_spin_lock+0x177/0x190 [ 8864.259290] _raw_spin_lock+0x21/0x30 [ 8864.260554] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8864.262665] process_one_work+0x2c8/0x700 [ 8864.264156] worker_thread+0x296/0x6e0 [ 8864.265360] ? rescuer_thread+0x570/0x570 [ 8864.267091] kthread+0x1d1/0x200 [ 8864.268074] ? set_kthread_struct+0x70/0x70 [ 8864.269935] ret_from_fork+0x1f/0x30 [ 8876.054035] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8876.058504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8876.083561] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8876.097899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8876.105717] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8876.108716] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8876.117741] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8876.121324] RAX: 0000000000002a0a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8876.124444] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8876.128191] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8876.131176] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8876.134270] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8876.137444] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8876.141516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8876.144334] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8876.152201] Call Trace: [ 8876.153474] [ 8876.154443] ? show_regs.cold.9+0x22/0x2f [ 8876.157004] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8876.159315] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8876.161343] ? watchdog+0x40/0x40 [ 8876.163049] ? hrtimer_interrupt+0x128/0x2c0 [ 8876.164257] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8876.167257] ? apic_timer_interrupt+0xf/0x20 [ 8876.169197] [ 8876.170247] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8876.172703] do_raw_spin_lock+0x177/0x190 [ 8876.174995] _raw_spin_lock+0x21/0x30 [ 8876.176993] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8876.182035] ? do_raw_spin_unlock+0x75/0x190 [ 8876.187973] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8876.191245] ? do_wait_intr+0xf0/0xf0 [ 8876.193269] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8876.196431] mdc_close+0x53b/0xd70 [mdc] [ 8876.198766] lmv_close+0x18d/0x490 [lmv] [ 8876.200888] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8876.204686] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8876.206407] ll_md_close+0x274/0x800 [lustre] [ 8876.214707] ll_file_release+0x128/0xc50 [lustre] [ 8876.222525] __fput+0x10d/0x440 [ 8876.224441] ____fput+0x12/0x20 [ 8876.228609] task_work_run+0xc6/0x110 [ 8876.230452] exit_to_usermode_loop+0x1dd/0x1f0 [ 8876.232932] do_syscall_64+0x3d6/0x440 [ 8876.238379] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8876.240239] RIP: 0033:0x7fdc0afbfcd8 [ 8876.242561] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8876.255395] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8876.259962] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8876.265720] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8876.271575] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8876.276401] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8876.280738] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8880.052022] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 8880.056936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8880.085894] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8880.093570] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8880.098735] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8880.100863] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8880.109470] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8880.111806] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8880.114205] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8880.116609] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8880.118948] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8880.121358] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8880.124489] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8880.127636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8880.129505] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8880.131727] Call Trace: [ 8880.132641] [ 8880.133222] ? show_regs.cold.9+0x22/0x2f [ 8880.134619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8880.136152] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8880.137656] ? watchdog+0x40/0x40 [ 8880.138785] ? hrtimer_interrupt+0x128/0x2c0 [ 8880.140151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8880.141915] ? apic_timer_interrupt+0xf/0x20 [ 8880.143510] [ 8880.144157] ? native_safe_halt+0x2e/0x50 [ 8880.146292] kvm_wait+0xcd/0xe0 [ 8880.147609] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8880.149407] do_raw_spin_lock+0x177/0x190 [ 8880.150898] _raw_spin_lock+0x21/0x30 [ 8880.152174] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8880.154150] ? do_raw_spin_unlock+0x75/0x190 [ 8880.155675] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8880.158164] ? do_wait_intr+0xf0/0xf0 [ 8880.159647] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8880.162100] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8880.166309] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8880.171533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8880.176083] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8880.181072] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8880.184680] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8880.189407] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8880.194522] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8880.200540] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8880.205354] lmv_intent_open+0x338/0x1150 [lmv] [ 8880.209164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8880.213573] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8880.217948] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8880.221920] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8880.225619] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8880.229806] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8880.234314] ? do_raw_spin_unlock+0x75/0x190 [ 8880.237639] ? _raw_spin_unlock+0x12/0x30 [ 8880.240187] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8880.242623] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8880.244877] ll_atomic_open+0x36a/0xd70 [lustre] [ 8880.248103] lookup_open+0x356/0xb40 [ 8880.249630] do_last+0x313/0xfc0 [ 8880.252938] ? nd_jump_root+0xe5/0x160 [ 8880.257288] ? path_init+0x437/0x520 [ 8880.258869] path_openat+0xf7/0x500 [ 8880.261615] ? do_raw_spin_unlock+0x75/0x190 [ 8880.266779] do_filp_open+0x99/0x140 [ 8880.269266] ? getname_flags+0x6e/0x330 [ 8880.272482] ? __check_object_size+0xff/0x256 [ 8880.277787] ? do_raw_spin_unlock+0x75/0x190 [ 8880.279607] ? _raw_spin_unlock+0x12/0x30 [ 8880.281408] do_sys_openat2+0x2b4/0x410 [ 8880.285310] do_sys_open+0x73/0xa0 [ 8880.287758] __x64_sys_openat+0x24/0x30 [ 8880.290879] do_syscall_64+0xc1/0x440 [ 8880.293792] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8880.298791] RIP: 0033:0x7fdc0afbf332 [ 8880.300303] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8880.314642] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8880.318800] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8880.324227] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8880.326561] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8880.328818] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8880.331318] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8884.055617] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8884.064608] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8884.115788] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8884.120887] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8884.131064] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8884.134901] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8884.149624] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8884.162459] RAX: 000000000000368c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8884.165555] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8884.175016] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8884.177161] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8884.180350] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8884.183437] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8884.185891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8884.187659] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8884.191560] Call Trace: [ 8884.192618] [ 8884.193646] ? show_regs.cold.9+0x22/0x2f [ 8884.195412] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8884.197729] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8884.200140] ? watchdog+0x40/0x40 [ 8884.201566] ? hrtimer_interrupt+0x128/0x2c0 [ 8884.204329] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8884.205552] ? apic_timer_interrupt+0xf/0x20 [ 8884.207161] [ 8884.207788] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8884.209806] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8884.212074] ? dump_stack+0xf6/0x10e [ 8884.213653] do_raw_spin_lock+0x177/0x190 [ 8884.215339] _raw_spin_lock+0x21/0x30 [ 8884.217038] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8884.219107] obd_mod_free+0x46/0x140 [ptlrpc] [ 8884.221526] mdc_commit_open+0x91/0xb0 [mdc] [ 8884.224289] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8884.227250] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8884.229972] after_reply+0xe2c/0x1780 [ptlrpc] [ 8884.232463] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8884.235061] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8884.237030] ? do_wait_intr+0xf0/0xf0 [ 8884.238297] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8884.242481] kthread+0x1d1/0x200 [ 8884.244344] ? set_kthread_struct+0x70/0x70 [ 8884.245771] ret_from_fork+0x1f/0x30 [ 8887.269301] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6862s! [ 8887.277062] Showing busy workqueues and worker pools: [ 8887.281727] workqueue events: flags=0x0 [ 8887.285614] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8887.296373] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8887.303085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8887.307034] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8887.317956] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8887.323283] pending: kfree_rcu_monitor [ 8887.325648] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8887.331825] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8887.347770] workqueue events_long: flags=0x0 [ 8887.354857] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8887.363550] pending: lu_objects_flush [obdclass] [ 8887.370884] workqueue events_unbound: flags=0x2 [ 8887.378546] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8887.384987] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8887.392754] workqueue events_power_efficient: flags=0x80 [ 8887.398220] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8887.405621] pending: neigh_periodic_work, check_lifetime [ 8887.408791] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8887.415542] pending: do_cache_clean, neigh_periodic_work [ 8887.422011] workqueue mm_percpu_wq: flags=0x8 [ 8887.424726] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8887.429258] pending: lru_add_drain_per_cpu [ 8887.431510] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8887.434729] pending: vmstat_update, lru_add_drain_per_cpu [ 8887.437297] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8887.440471] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8887.443352] workqueue writeback: flags=0x4a [ 8887.445193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8887.447790] pending: wb_workfn, wb_workfn [ 8887.449789] workqueue xprtiod: flags=0xa [ 8887.451368] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8887.454082] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8887.460350] workqueue ipv6_addrconf: flags=0x40008 [ 8887.464851] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8887.467618] pending: addrconf_verify_work [ 8887.470255] workqueue knbd0-recv: flags=0x1a [ 8887.473540] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8887.478730] in-flight: 655:recv_work [ 8887.483796] workqueue ptlrpc_pinger: flags=0x2 [ 8887.487751] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8887.493102] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8887.500535] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6862s workers=4 idle: 36396 38 11188 [ 8887.508355] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8878s workers=2 idle: 658 [ 8892.051033] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8892.054056] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8892.077458] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8892.082266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8892.087713] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8892.090827] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8892.093177] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8892.102089] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8892.105637] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8892.109223] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8892.112705] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8892.116266] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8892.119971] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8892.123472] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8892.127048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8892.130548] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8892.133995] Call Trace: [ 8892.135420] [ 8892.136585] ? show_regs.cold.9+0x22/0x2f [ 8892.138670] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8892.143836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8892.146264] ? watchdog+0x40/0x40 [ 8892.148060] ? hrtimer_interrupt+0x128/0x2c0 [ 8892.150199] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8892.152741] ? apic_timer_interrupt+0xf/0x20 [ 8892.154817] [ 8892.155779] ? native_safe_halt+0x2e/0x50 [ 8892.158291] kvm_wait+0xcd/0xe0 [ 8892.159684] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8892.162126] do_raw_spin_lock+0x177/0x190 [ 8892.164285] _raw_spin_lock+0x21/0x30 [ 8892.166160] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8892.169146] process_one_work+0x2c8/0x700 [ 8892.170857] worker_thread+0x296/0x6e0 [ 8892.172817] ? rescuer_thread+0x570/0x570 [ 8892.174616] kthread+0x1d1/0x200 [ 8892.176366] ? set_kthread_struct+0x70/0x70 [ 8892.178459] ret_from_fork+0x1f/0x30 [ 8904.054009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8904.058141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8904.078563] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8904.083469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8904.087094] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8904.090046] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8904.098812] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8904.102920] RAX: 0000000000004f70 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8904.107321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8904.112086] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8904.116015] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8904.119826] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8904.122617] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8904.126846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8904.129090] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8904.133859] Call Trace: [ 8904.135321] [ 8904.137505] ? show_regs.cold.9+0x22/0x2f [ 8904.142318] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8904.145397] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8904.147616] ? watchdog+0x40/0x40 [ 8904.148906] ? hrtimer_interrupt+0x128/0x2c0 [ 8904.150508] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8904.152635] ? apic_timer_interrupt+0xf/0x20 [ 8904.154568] [ 8904.155817] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8904.158082] do_raw_spin_lock+0x177/0x190 [ 8904.159563] _raw_spin_lock+0x21/0x30 [ 8904.160859] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8904.163442] ? do_raw_spin_unlock+0x75/0x190 [ 8904.165260] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8904.167339] ? do_wait_intr+0xf0/0xf0 [ 8904.168732] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8904.171521] mdc_close+0x53b/0xd70 [mdc] [ 8904.173185] lmv_close+0x18d/0x490 [lmv] [ 8904.174733] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8904.177058] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8904.179594] ll_md_close+0x274/0x800 [lustre] [ 8904.182156] ll_file_release+0x128/0xc50 [lustre] [ 8904.185214] __fput+0x10d/0x440 [ 8904.187869] ____fput+0x12/0x20 [ 8904.191319] task_work_run+0xc6/0x110 [ 8904.193459] exit_to_usermode_loop+0x1dd/0x1f0 [ 8904.195860] do_syscall_64+0x3d6/0x440 [ 8904.197372] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8904.200164] RIP: 0033:0x7fdc0afbfcd8 [ 8904.201550] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8904.211086] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8904.213818] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8904.216428] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8904.219079] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8904.221765] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8904.224310] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8908.053795] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 8908.062471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8908.110767] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8908.124955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8908.137195] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8908.144243] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8908.158020] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8908.164434] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8908.171347] RDX: 0000000000001980 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8908.183343] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8908.193075] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8908.198831] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8908.205188] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8908.208664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8908.211224] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8908.217469] Call Trace: [ 8908.219832] [ 8908.220496] ? show_regs.cold.9+0x22/0x2f [ 8908.226597] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8908.228372] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8908.232327] ? watchdog+0x40/0x40 [ 8908.235148] ? hrtimer_interrupt+0x128/0x2c0 [ 8908.239333] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8908.250114] ? apic_timer_interrupt+0xf/0x20 [ 8908.260145] [ 8908.264107] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8908.274809] do_raw_spin_lock+0x177/0x190 [ 8908.278435] _raw_spin_lock+0x21/0x30 [ 8908.287698] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8908.301217] ? do_raw_spin_unlock+0x75/0x190 [ 8908.305194] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8908.310949] ? do_wait_intr+0xf0/0xf0 [ 8908.316891] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8908.321895] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8908.325212] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8908.328841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.332944] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8908.336753] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8908.340968] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8908.348210] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.360206] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8908.370259] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8908.378445] lmv_intent_open+0x338/0x1150 [lmv] [ 8908.383837] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.387294] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.392142] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8908.395790] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8908.400815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.407399] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8908.414277] ? do_raw_spin_unlock+0x75/0x190 [ 8908.417793] ? _raw_spin_unlock+0x12/0x30 [ 8908.419437] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8908.425506] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8908.429339] ll_atomic_open+0x36a/0xd70 [lustre] [ 8908.431858] lookup_open+0x356/0xb40 [ 8908.439469] do_last+0x313/0xfc0 [ 8908.444271] ? nd_jump_root+0xe5/0x160 [ 8908.447401] ? path_init+0x437/0x520 [ 8908.451013] path_openat+0xf7/0x500 [ 8908.453471] ? do_raw_spin_unlock+0x75/0x190 [ 8908.456529] do_filp_open+0x99/0x140 [ 8908.461094] ? getname_flags+0x6e/0x330 [ 8908.467911] ? __check_object_size+0xff/0x256 [ 8908.472326] ? do_raw_spin_unlock+0x75/0x190 [ 8908.480192] ? _raw_spin_unlock+0x12/0x30 [ 8908.488318] do_sys_openat2+0x2b4/0x410 [ 8908.494045] do_sys_open+0x73/0xa0 [ 8908.495408] __x64_sys_openat+0x24/0x30 [ 8908.497475] do_syscall_64+0xc1/0x440 [ 8908.499191] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8908.502393] RIP: 0033:0x7fdc0afbf332 [ 8908.504253] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8908.525276] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8908.531395] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8908.536246] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8908.548368] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8908.553134] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8908.556732] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8912.048030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8912.050733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8912.067778] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8912.072302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8912.075565] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8912.077934] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8912.085298] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8912.088371] RAX: 0000000000006edd RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8912.091140] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8912.093798] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8912.097330] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8912.099388] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8912.102127] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8912.104790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8912.106851] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8912.109926] Call Trace: [ 8912.110738] [ 8912.111514] ? show_regs.cold.9+0x22/0x2f [ 8912.112839] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8912.114562] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8912.116331] ? watchdog+0x40/0x40 [ 8912.117507] ? hrtimer_interrupt+0x128/0x2c0 [ 8912.118909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8912.120603] ? apic_timer_interrupt+0xf/0x20 [ 8912.122146] [ 8912.122791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8912.124723] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8912.126534] ? dump_stack+0xf6/0x10e [ 8912.127780] do_raw_spin_lock+0x177/0x190 [ 8912.129249] _raw_spin_lock+0x21/0x30 [ 8912.130466] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8912.132348] obd_mod_free+0x46/0x140 [ptlrpc] [ 8912.134148] mdc_commit_open+0x91/0xb0 [mdc] [ 8912.135746] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8912.138047] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8912.140329] after_reply+0xe2c/0x1780 [ptlrpc] [ 8912.142106] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8912.144212] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8912.146469] ? do_wait_intr+0xf0/0xf0 [ 8912.147776] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8912.149775] kthread+0x1d1/0x200 [ 8912.150917] ? set_kthread_struct+0x70/0x70 [ 8912.152390] ret_from_fork+0x1f/0x30 [ 8917.983889] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6893s! [ 8917.992900] Showing busy workqueues and worker pools: [ 8917.994570] workqueue events: flags=0x0 [ 8917.996184] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8918.000128] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8918.010216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8918.014824] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8918.025414] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8918.028998] pending: kfree_rcu_monitor [ 8918.033179] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8918.039942] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8918.050394] workqueue events_long: flags=0x0 [ 8918.052807] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8918.060272] pending: lu_objects_flush [obdclass] [ 8918.062982] workqueue events_unbound: flags=0x2 [ 8918.065851] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8918.076417] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8918.086992] workqueue events_power_efficient: flags=0x80 [ 8918.094919] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8918.104823] pending: neigh_periodic_work, check_lifetime [ 8918.111816] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8918.116897] pending: do_cache_clean, neigh_periodic_work [ 8918.123040] workqueue mm_percpu_wq: flags=0x8 [ 8918.139752] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8918.143220] pending: lru_add_drain_per_cpu [ 8918.146044] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8918.151895] pending: vmstat_update, lru_add_drain_per_cpu [ 8918.159980] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8918.169376] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8918.172595] workqueue writeback: flags=0x4a [ 8918.177996] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8918.182224] pending: wb_workfn, wb_workfn [ 8918.187032] workqueue xprtiod: flags=0xa [ 8918.190665] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8918.198941] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8918.210206] workqueue ipv6_addrconf: flags=0x40008 [ 8918.220648] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8918.226049] pending: addrconf_verify_work [ 8918.231438] workqueue knbd0-recv: flags=0x1a [ 8918.233210] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8918.236253] in-flight: 655:recv_work [ 8918.238108] workqueue ptlrpc_pinger: flags=0x2 [ 8918.239667] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8918.241706] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8918.243893] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6893s workers=4 idle: 36396 38 11188 [ 8918.249115] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8908s workers=2 idle: 658 [ 8919.507025] rcu: INFO: rcu_sched self-detected stall on CPU [ 8919.510059] rcu: 0-...!: (4320442 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 8919.514561] (t=6900164 jiffies g=270761 q=4389) [ 8919.516654] rcu: rcu_sched kthread starved for 6895096 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 8919.520073] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8919.523984] rcu: RCU grace-period kthread stack dump: [ 8919.527110] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8919.537410] Call Trace: [ 8919.539691] __schedule+0x351/0xcb0 [ 8919.542223] schedule+0xc0/0x180 [ 8919.543480] schedule_timeout+0xb4/0x190 [ 8919.544621] ? do_raw_spin_unlock+0x75/0x190 [ 8919.545864] ? __next_timer_interrupt+0x160/0x160 [ 8919.547455] ? prepare_to_swait_event+0xf3/0x140 [ 8919.548908] rcu_gp_kthread+0x8dd/0x15b0 [ 8919.550237] ? force_qs_rnp+0x2c0/0x2c0 [ 8919.551609] kthread+0x1d1/0x200 [ 8919.552867] ? set_kthread_struct+0x70/0x70 [ 8919.554493] ret_from_fork+0x1f/0x30 [ 8919.556121] rcu: Stack dump where RCU GP kthread last ran: [ 8919.557909] Sending NMI from CPU 0 to CPUs 2: [ 8919.566457] NMI backtrace for cpu 2 [ 8919.566465] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8919.566468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8919.566469] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8919.566471] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8919.566473] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 [ 8919.566476] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8919.566477] RDX: 0000000000006ddc RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8919.566479] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8919.566480] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8919.566481] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8919.566482] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8919.566484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8919.566485] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8919.566486] Call Trace: [ 8919.566487] [ 8919.566488] ? show_regs.cold.9+0x22/0x2f [ 8919.566489] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8919.566490] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8919.566491] ? nmi_handle+0xbd/0x210 [ 8919.566492] ? default_do_nmi+0x4b/0x1b0 [ 8919.566493] ? do_nmi+0x280/0x380 [ 8919.566494] ? end_repeat_nmi+0x16/0x1f [ 8919.566495] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8919.566496] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8919.566498] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8919.566499] [ 8919.566500] do_raw_spin_lock+0x177/0x190 [ 8919.566501] _raw_spin_lock+0x21/0x30 [ 8919.566502] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8919.566503] ? do_raw_spin_unlock+0x75/0x190 [ 8919.566504] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8919.566505] ? do_wait_intr+0xf0/0xf0 [ 8919.566507] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8919.566508] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8919.566509] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8919.566510] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8919.566512] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8919.566513] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8919.566514] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8919.566515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8919.566517] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8919.566518] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8919.566519] lmv_intent_open+0x338/0x1150 [lmv] [ 8919.566520] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8919.566521] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8919.566522] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8919.566523] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8919.566524] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8919.566525] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8919.566526] ? do_raw_spin_unlock+0x75/0x190 [ 8919.566527] ? _raw_spin_unlock+0x12/0x30 [ 8919.566528] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8919.566529] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8919.566530] ll_atomic_open+0x36a/0xd70 [lustre] [ 8919.566531] lookup_open+0x356/0xb40 [ 8919.566532] do_last+0x313/0xfc0 [ 8919.566533] ? nd_jump_root+0xe5/0x160 [ 8919.566535] ? path_init+0x437/0x520 [ 8919.566535] path_openat+0xf7/0x500 [ 8919.566536] ? do_raw_spin_unlock+0x75/0x190 [ 8919.566538] do_filp_open+0x99/0x140 [ 8919.566539] ? getname_flags+0x6e/0x330 [ 8919.566540] ? __check_object_size+0xff/0x256 [ 8919.566541] ? do_raw_spin_unlock+0x75/0x190 [ 8919.566542] ? _raw_spin_unlock+0x12/0x30 [ 8919.566543] do_sys_openat2+0x2b4/0x410 [ 8919.566544] do_sys_open+0x73/0xa0 [ 8919.566545] __x64_sys_openat+0x24/0x30 [ 8919.566546] do_syscall_64+0xc1/0x440 [ 8919.566547] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8919.566548] RIP: 0033:0x7fdc0afbf332 [ 8919.566550] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8919.566552] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8919.566554] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8919.566556] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8919.566557] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8919.566558] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8919.566560] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8919.567106] NMI backtrace for cpu 0 [ 8919.865213] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8919.870347] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8919.873946] Call Trace: [ 8919.874782] [ 8919.875558] ? dump_stack+0xbb/0x10e [ 8919.877038] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8919.878802] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8919.880185] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8919.881923] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8919.883591] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8919.885173] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8919.886548] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8919.888034] ? update_process_times+0x6d/0xc0 [ 8919.889444] ? tick_sched_handle.isra.18+0x27/0x90 [ 8919.891409] ? tick_sched_timer+0x53/0xb0 [ 8919.892946] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8919.894541] ? tick_sched_handle.isra.18+0x90/0x90 [ 8919.896527] ? hrtimer_interrupt+0x128/0x2c0 [ 8919.898585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8919.908624] ? apic_timer_interrupt+0xf/0x20 [ 8919.911062] [ 8919.912795] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8919.917228] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8919.923732] ? dump_stack+0xf6/0x10e [ 8919.925751] ? do_raw_spin_lock+0x177/0x190 [ 8919.928343] ? _raw_spin_lock+0x21/0x30 [ 8919.930364] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8919.933103] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8919.937228] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8919.939474] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8919.944159] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8919.947228] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8919.950790] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8919.954473] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8919.957735] ? do_wait_intr+0xf0/0xf0 [ 8919.960216] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8919.968869] ? kthread+0x1d1/0x200 [ 8919.972550] ? set_kthread_struct+0x70/0x70 [ 8919.976100] ? ret_from_fork+0x1f/0x30 [ 8920.052199] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8920.055558] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8920.126667] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8920.138250] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8920.147056] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8920.156579] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8920.162221] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8920.173362] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8920.180127] RAX: 00000000000073a2 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8920.183811] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8920.187893] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8920.197748] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8920.206495] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8920.213713] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8920.219508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8920.222594] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8920.226132] Call Trace: [ 8920.228677] [ 8920.229419] ? show_regs.cold.9+0x22/0x2f [ 8920.231738] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8920.233828] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8920.236580] ? watchdog+0x40/0x40 [ 8920.238209] ? hrtimer_interrupt+0x128/0x2c0 [ 8920.239840] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8920.242100] ? apic_timer_interrupt+0xf/0x20 [ 8920.243790] [ 8920.244578] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8920.246759] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8920.249187] do_raw_spin_lock+0x177/0x190 [ 8920.250987] _raw_spin_lock+0x21/0x30 [ 8920.252518] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8920.254893] process_one_work+0x2c8/0x700 [ 8920.256635] worker_thread+0x296/0x6e0 [ 8920.257893] ? rescuer_thread+0x570/0x570 [ 8920.259608] kthread+0x1d1/0x200 [ 8920.261424] ? set_kthread_struct+0x70/0x70 [ 8920.266166] ret_from_fork+0x1f/0x30 [ 8932.056351] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8932.062423] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8932.097452] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8932.102960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8932.107349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8932.113364] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8932.123299] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8932.128554] RAX: 0000000000007f1d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8932.141412] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8932.148233] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8932.155619] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8932.164332] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8932.173343] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8932.182055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8932.188657] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8932.196221] Call Trace: [ 8932.197445] [ 8932.198899] ? show_regs.cold.9+0x22/0x2f [ 8932.200748] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8932.203228] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8932.205764] ? watchdog+0x40/0x40 [ 8932.207278] ? hrtimer_interrupt+0x128/0x2c0 [ 8932.209264] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8932.211387] ? apic_timer_interrupt+0xf/0x20 [ 8932.213393] [ 8932.214346] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8932.220793] do_raw_spin_lock+0x177/0x190 [ 8932.222466] _raw_spin_lock+0x21/0x30 [ 8932.225844] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8932.229627] ? do_raw_spin_unlock+0x75/0x190 [ 8932.234746] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8932.242261] ? do_wait_intr+0xf0/0xf0 [ 8932.245353] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8932.249652] mdc_close+0x53b/0xd70 [mdc] [ 8932.252039] lmv_close+0x18d/0x490 [lmv] [ 8932.254003] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8932.257018] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8932.259370] ll_md_close+0x274/0x800 [lustre] [ 8932.263070] ll_file_release+0x128/0xc50 [lustre] [ 8932.266309] __fput+0x10d/0x440 [ 8932.269177] ____fput+0x12/0x20 [ 8932.270871] task_work_run+0xc6/0x110 [ 8932.272553] exit_to_usermode_loop+0x1dd/0x1f0 [ 8932.274405] do_syscall_64+0x3d6/0x440 [ 8932.276171] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8932.278653] RIP: 0033:0x7fdc0afbfcd8 [ 8932.280413] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8932.293033] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8932.296547] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8932.300226] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8932.304924] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8932.311435] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8932.318169] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8944.048034] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8944.050820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8944.052058] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8944.091533] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8944.095227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8944.095313] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8944.095321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8944.095324] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8944.095337] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8944.095342] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8944.095346] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8944.095349] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8944.095351] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8944.095353] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8944.095356] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8944.095359] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8944.095363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8944.095365] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8944.095374] Call Trace: [ 8944.095380] [ 8944.095385] ? show_regs.cold.9+0x22/0x2f [ 8944.095396] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8944.095402] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8944.095407] ? watchdog+0x40/0x40 [ 8944.095411] ? hrtimer_interrupt+0x128/0x2c0 [ 8944.095416] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8944.095424] ? apic_timer_interrupt+0xf/0x20 [ 8944.095429] [ 8944.095432] ? native_safe_halt+0x2e/0x50 [ 8944.095436] kvm_wait+0xcd/0xe0 [ 8944.095444] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8944.095452] do_raw_spin_lock+0x177/0x190 [ 8944.095456] _raw_spin_lock+0x21/0x30 [ 8944.095462] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8944.095723] ? do_raw_spin_unlock+0x75/0x190 [ 8944.095729] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8944.105048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8944.105059] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8944.105072] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8944.105076] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8944.105082] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8944.105084] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8944.105086] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8944.105089] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8944.105091] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8944.105095] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8944.105098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8944.105100] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8944.105109] Call Trace: [ 8944.105114] [ 8944.105120] ? show_regs.cold.9+0x22/0x2f [ 8944.105131] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8944.105139] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8944.105142] ? watchdog+0x40/0x40 [ 8944.105146] ? hrtimer_interrupt+0x128/0x2c0 [ 8944.105150] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8944.105157] ? apic_timer_interrupt+0xf/0x20 [ 8944.105162] [ 8944.105165] ? native_safe_halt+0x2e/0x50 [ 8944.105169] kvm_wait+0xcd/0xe0 [ 8944.105176] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8944.105182] ? dump_stack+0xf6/0x10e [ 8944.105188] do_raw_spin_lock+0x177/0x190 [ 8944.105192] _raw_spin_lock+0x21/0x30 [ 8944.105198] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8944.105468] obd_mod_free+0x46/0x140 [ptlrpc] [ 8944.105673] mdc_commit_open+0x91/0xb0 [mdc] [ 8944.105709] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8944.108169] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8944.108388] after_reply+0xe2c/0x1780 [ptlrpc] [ 8944.108583] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8944.108790] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8944.177227] ? do_wait_intr+0xf0/0xf0 [ 8944.177248] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8944.177533] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8944.177742] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8944.177952] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8944.178064] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8944.178105] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8944.178136] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8944.178167] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8944.178251] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8944.178477] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8944.178506] lmv_intent_open+0x338/0x1150 [lmv] [ 8944.178537] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8944.178620] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8944.178706] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8944.178728] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8944.178812] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8944.178895] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8944.178979] ? do_raw_spin_unlock+0x75/0x190 [ 8944.178986] ? _raw_spin_unlock+0x12/0x30 [ 8944.178992] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8944.179022] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8944.179047] ll_atomic_open+0x36a/0xd70 [lustre] [ 8944.179133] lookup_open+0x356/0xb40 [ 8944.179142] do_last+0x313/0xfc0 [ 8944.179147] ? nd_jump_root+0xe5/0x160 [ 8944.179151] ? path_init+0x437/0x520 [ 8944.179156] path_openat+0xf7/0x500 [ 8944.179160] ? do_raw_spin_unlock+0x75/0x190 [ 8944.179165] do_filp_open+0x99/0x140 [ 8944.179171] ? getname_flags+0x6e/0x330 [ 8944.179175] ? __check_object_size+0xff/0x256 [ 8944.179180] ? do_raw_spin_unlock+0x75/0x190 [ 8944.179184] ? _raw_spin_unlock+0x12/0x30 [ 8944.179188] do_sys_openat2+0x2b4/0x410 [ 8944.179192] do_sys_open+0x73/0xa0 [ 8944.179196] __x64_sys_openat+0x24/0x30 [ 8944.179199] do_syscall_64+0xc1/0x440 [ 8944.179206] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8944.179211] RIP: 0033:0x7fdc0afbf332 [ 8944.179217] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8944.179221] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8944.179227] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8944.179229] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8944.179232] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8944.179234] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8944.179237] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8944.467166] ? do_wait_intr+0xf0/0xf0 [ 8944.468770] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8944.471152] kthread+0x1d1/0x200 [ 8944.472546] ? set_kthread_struct+0x70/0x70 [ 8944.473987] ret_from_fork+0x1f/0x30 [ 8948.051056] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8948.054570] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8948.082225] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8948.087366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8948.095371] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8948.101303] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8948.103806] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8948.116496] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8948.120775] RAX: 000000000000041b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8948.125859] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8948.131187] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8948.136556] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8948.142624] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8948.149007] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8948.161048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8948.170171] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8948.178744] Call Trace: [ 8948.181517] [ 8948.187549] ? show_regs.cold.9+0x22/0x2f [ 8948.195303] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8948.207228] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8948.208760] ? watchdog+0x40/0x40 [ 8948.210181] ? hrtimer_interrupt+0x128/0x2c0 [ 8948.212002] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8948.213913] ? apic_timer_interrupt+0xf/0x20 [ 8948.215534] [ 8948.216310] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8948.218217] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8948.220153] do_raw_spin_lock+0x177/0x190 [ 8948.221637] _raw_spin_lock+0x21/0x30 [ 8948.223007] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8948.225278] process_one_work+0x2c8/0x700 [ 8948.226619] worker_thread+0x296/0x6e0 [ 8948.228070] ? rescuer_thread+0x570/0x570 [ 8948.229675] kthread+0x1d1/0x200 [ 8948.231042] ? set_kthread_struct+0x70/0x70 [ 8948.232407] ret_from_fork+0x1f/0x30 [ 8948.703049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6923s! [ 8948.709683] Showing busy workqueues and worker pools: [ 8948.711592] workqueue events: flags=0x0 [ 8948.713375] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8948.717055] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8948.725466] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8948.733532] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8948.740738] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8948.743653] pending: kfree_rcu_monitor [ 8948.745446] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8948.749085] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8948.759608] workqueue events_long: flags=0x0 [ 8948.761613] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8948.766044] pending: lu_objects_flush [obdclass] [ 8948.768023] workqueue events_unbound: flags=0x2 [ 8948.769640] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8948.772093] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8948.779030] workqueue events_power_efficient: flags=0x80 [ 8948.783034] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8948.785525] pending: neigh_periodic_work, check_lifetime [ 8948.788247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8948.790681] pending: do_cache_clean, neigh_periodic_work [ 8948.792882] workqueue mm_percpu_wq: flags=0x8 [ 8948.794978] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8948.797655] pending: lru_add_drain_per_cpu [ 8948.799559] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8948.803685] pending: vmstat_update, lru_add_drain_per_cpu [ 8948.808381] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8948.813223] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8948.818735] workqueue writeback: flags=0x4a [ 8948.821494] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8948.827406] pending: wb_workfn, wb_workfn [ 8948.829435] workqueue xprtiod: flags=0xa [ 8948.831676] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8948.841647] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8948.849133] workqueue ipv6_addrconf: flags=0x40008 [ 8948.854027] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8948.857594] pending: addrconf_verify_work [ 8948.865018] workqueue knbd0-recv: flags=0x1a [ 8948.870081] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8948.873555] in-flight: 655:recv_work [ 8948.875037] workqueue ptlrpc_pinger: flags=0x2 [ 8948.876736] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8948.878696] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8948.884471] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6923s workers=4 idle: 36396 38 11188 [ 8948.890099] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8939s workers=2 idle: 658 [ 8960.058337] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8960.064294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8960.087735] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8960.092954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8960.100176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8960.105287] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8960.121886] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8960.125418] RAX: 0000000000004120 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8960.129562] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8960.132768] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8960.135963] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8960.138722] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8960.141461] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8960.144274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8960.145660] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8960.147369] Call Trace: [ 8960.148019] [ 8960.148488] ? show_regs.cold.9+0x22/0x2f [ 8960.149461] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8960.150736] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8960.151856] ? watchdog+0x40/0x40 [ 8960.152757] ? hrtimer_interrupt+0x128/0x2c0 [ 8960.154531] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8960.156787] ? apic_timer_interrupt+0xf/0x20 [ 8960.158752] [ 8960.159754] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8960.162306] do_raw_spin_lock+0x177/0x190 [ 8960.164439] _raw_spin_lock+0x21/0x30 [ 8960.167137] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8960.169959] ? do_raw_spin_unlock+0x75/0x190 [ 8960.172432] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8960.175117] ? do_wait_intr+0xf0/0xf0 [ 8960.176909] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8960.179559] mdc_close+0x53b/0xd70 [mdc] [ 8960.181483] lmv_close+0x18d/0x490 [lmv] [ 8960.183367] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8960.185962] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8960.188210] ll_md_close+0x274/0x800 [lustre] [ 8960.190336] ll_file_release+0x128/0xc50 [lustre] [ 8960.192435] __fput+0x10d/0x440 [ 8960.194326] ____fput+0x12/0x20 [ 8960.195683] task_work_run+0xc6/0x110 [ 8960.197280] exit_to_usermode_loop+0x1dd/0x1f0 [ 8960.199109] do_syscall_64+0x3d6/0x440 [ 8960.200538] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8960.202781] RIP: 0033:0x7fdc0afbfcd8 [ 8960.204334] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8960.212091] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8960.215329] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8960.218313] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8960.221393] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8960.224645] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8960.227850] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8972.055384] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 8972.062752] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8972.104353] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8972.110777] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8972.116824] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8972.119323] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8972.140366] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8972.151586] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 8972.159777] RDX: 00000000000079ca RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 8972.168654] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 8972.173935] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 8972.181873] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8972.184212] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 8972.195883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8972.202398] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 8972.209435] Call Trace: [ 8972.211561] [ 8972.214012] ? show_regs.cold.9+0x22/0x2f [ 8972.219584] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8972.223609] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8972.225851] ? watchdog+0x40/0x40 [ 8972.226952] ? hrtimer_interrupt+0x128/0x2c0 [ 8972.233227] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8972.239198] ? apic_timer_interrupt+0xf/0x20 [ 8972.244742] [ 8972.247200] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8972.253061] do_raw_spin_lock+0x177/0x190 [ 8972.255808] _raw_spin_lock+0x21/0x30 [ 8972.258344] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8972.264463] ? do_raw_spin_unlock+0x75/0x190 [ 8972.270744] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8972.275821] ? do_wait_intr+0xf0/0xf0 [ 8972.277644] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8972.282461] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8972.285763] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8972.289234] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8972.294166] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8972.297063] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8972.299122] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8972.304275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8972.306761] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8972.310018] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8972.313441] lmv_intent_open+0x338/0x1150 [lmv] [ 8972.315619] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8972.319227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8972.321532] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8972.327643] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8972.331530] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8972.333814] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8972.337907] ? do_raw_spin_unlock+0x75/0x190 [ 8972.343216] ? _raw_spin_unlock+0x12/0x30 [ 8972.348815] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8972.354192] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8972.361009] ll_atomic_open+0x36a/0xd70 [lustre] [ 8972.367190] lookup_open+0x356/0xb40 [ 8972.372272] do_last+0x313/0xfc0 [ 8972.375665] ? nd_jump_root+0xe5/0x160 [ 8972.379091] ? path_init+0x437/0x520 [ 8972.380654] path_openat+0xf7/0x500 [ 8972.382822] ? do_raw_spin_unlock+0x75/0x190 [ 8972.384966] do_filp_open+0x99/0x140 [ 8972.386560] ? getname_flags+0x6e/0x330 [ 8972.389582] ? __check_object_size+0xff/0x256 [ 8972.391072] ? do_raw_spin_unlock+0x75/0x190 [ 8972.398100] ? _raw_spin_unlock+0x12/0x30 [ 8972.402309] do_sys_openat2+0x2b4/0x410 [ 8972.408346] do_sys_open+0x73/0xa0 [ 8972.410547] __x64_sys_openat+0x24/0x30 [ 8972.411419] do_syscall_64+0xc1/0x440 [ 8972.412852] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8972.414512] RIP: 0033:0x7fdc0afbf332 [ 8972.415762] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8972.428933] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8972.435901] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 8972.441330] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 8972.446100] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 8972.450570] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 8972.454817] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 8976.049686] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 8976.053843] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 8976.054621] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8976.054706] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8976.054776] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8976.054780] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8976.054796] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8976.054801] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8976.054805] RAX: 0000000000007f87 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8976.054808] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 8976.054810] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 8976.054813] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 8976.054815] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 8976.054818] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 8976.054821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8976.054824] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 8976.054833] Call Trace: [ 8976.054839] [ 8976.054846] ? show_regs.cold.9+0x22/0x2f [ 8976.054857] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8976.054864] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8976.054869] ? watchdog+0x40/0x40 [ 8976.054874] ? hrtimer_interrupt+0x128/0x2c0 [ 8976.054878] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8976.054887] ? apic_timer_interrupt+0xf/0x20 [ 8976.054892] [ 8976.054895] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8976.054900] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8976.054904] ? dump_stack+0xf6/0x10e [ 8976.054911] do_raw_spin_lock+0x177/0x190 [ 8976.054916] _raw_spin_lock+0x21/0x30 [ 8976.054922] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8976.055302] obd_mod_free+0x46/0x140 [ptlrpc] [ 8976.066122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8976.066214] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8976.066225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8976.066236] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8976.099535] mdc_commit_open+0x91/0xb0 [mdc] [ 8976.099590] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8976.099891] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8976.100096] after_reply+0xe2c/0x1780 [ptlrpc] [ 8976.100296] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8976.100644] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8976.100859] ? do_wait_intr+0xf0/0xf0 [ 8976.100870] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8976.114001] [ 8976.114012] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8976.114030] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8976.114050] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8976.114056] RAX: 0000000000007e85 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8976.114059] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 8976.114061] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8976.114063] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 8976.114066] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 8976.114069] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 8976.114072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8976.114075] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 8976.114085] Call Trace: [ 8976.114091] [ 8976.114097] ? show_regs.cold.9+0x22/0x2f [ 8976.114106] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8976.114112] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8976.114117] ? watchdog+0x40/0x40 [ 8976.114121] ? hrtimer_interrupt+0x128/0x2c0 [ 8976.114124] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8976.114132] ? apic_timer_interrupt+0xf/0x20 [ 8976.114137] [ 8976.114140] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8976.114145] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8976.114153] do_raw_spin_lock+0x177/0x190 [ 8976.114159] _raw_spin_lock+0x21/0x30 [ 8976.114164] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8976.123468] kthread+0x1d1/0x200 [ 8976.123493] ? set_kthread_struct+0x70/0x70 [ 8976.123498] ret_from_fork+0x1f/0x30 [ 8976.339243] process_one_work+0x2c8/0x700 [ 8976.340983] worker_thread+0x296/0x6e0 [ 8976.343134] ? rescuer_thread+0x570/0x570 [ 8976.345165] kthread+0x1d1/0x200 [ 8976.346745] ? set_kthread_struct+0x70/0x70 [ 8976.348490] ret_from_fork+0x1f/0x30 [ 8978.911095] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6953s! [ 8978.916186] Showing busy workqueues and worker pools: [ 8978.918562] workqueue events: flags=0x0 [ 8978.919650] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8978.923931] pending: cpu_latency_work [ptlrpc], push_to_pool [ 8978.926689] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8978.935718] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 8978.950022] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8978.953680] pending: kfree_rcu_monitor [ 8978.955621] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8978.973607] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 8978.987846] workqueue events_long: flags=0x0 [ 8978.990604] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8978.998749] pending: lu_objects_flush [obdclass] [ 8979.005702] workqueue events_unbound: flags=0x2 [ 8979.015091] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8979.020909] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8979.034239] workqueue events_power_efficient: flags=0x80 [ 8979.037231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8979.040467] pending: neigh_periodic_work, check_lifetime [ 8979.045062] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8979.047754] pending: do_cache_clean, neigh_periodic_work [ 8979.050497] workqueue mm_percpu_wq: flags=0x8 [ 8979.052531] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8979.055562] pending: lru_add_drain_per_cpu [ 8979.057560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8979.060669] pending: vmstat_update, lru_add_drain_per_cpu [ 8979.064551] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8979.067750] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8979.070811] workqueue writeback: flags=0x4a [ 8979.072968] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8979.076041] pending: wb_workfn, wb_workfn [ 8979.078160] workqueue xprtiod: flags=0xa [ 8979.080044] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8979.083207] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8979.087789] workqueue ipv6_addrconf: flags=0x40008 [ 8979.090205] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8979.093542] pending: addrconf_verify_work [ 8979.096568] workqueue knbd0-recv: flags=0x1a [ 8979.098576] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8979.101593] in-flight: 655:recv_work [ 8979.103956] workqueue ptlrpc_pinger: flags=0x2 [ 8979.106157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8979.109230] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 8979.112674] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6954s workers=4 idle: 36396 38 11188 [ 8979.117109] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8969s workers=2 idle: 658 [ 8988.054551] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 8988.058079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8988.080243] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8988.089854] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8988.093558] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8988.095441] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8988.101830] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8988.104753] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 8988.107416] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 8988.109769] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8988.112328] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 8988.114952] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 8988.118583] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 8988.128311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8988.132635] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 8988.138489] Call Trace: [ 8988.139520] [ 8988.140330] ? show_regs.cold.9+0x22/0x2f [ 8988.142614] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8988.144718] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8988.146974] ? watchdog+0x40/0x40 [ 8988.148349] ? hrtimer_interrupt+0x128/0x2c0 [ 8988.149949] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8988.151679] ? apic_timer_interrupt+0xf/0x20 [ 8988.153587] [ 8988.154564] ? native_safe_halt+0x2e/0x50 [ 8988.156412] kvm_wait+0xcd/0xe0 [ 8988.157459] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8988.159562] do_raw_spin_lock+0x177/0x190 [ 8988.160795] _raw_spin_lock+0x21/0x30 [ 8988.161937] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8988.164261] ? do_raw_spin_unlock+0x75/0x190 [ 8988.165891] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8988.167919] ? do_wait_intr+0xf0/0xf0 [ 8988.169323] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8988.171480] mdc_close+0x53b/0xd70 [mdc] [ 8988.172921] lmv_close+0x18d/0x490 [lmv] [ 8988.174423] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8988.176729] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8988.179580] ll_md_close+0x274/0x800 [lustre] [ 8988.181314] ll_file_release+0x128/0xc50 [lustre] [ 8988.183309] __fput+0x10d/0x440 [ 8988.184255] ____fput+0x12/0x20 [ 8988.187793] task_work_run+0xc6/0x110 [ 8988.189725] exit_to_usermode_loop+0x1dd/0x1f0 [ 8988.191270] do_syscall_64+0x3d6/0x440 [ 8988.194708] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8988.200075] RIP: 0033:0x7fdc0afbfcd8 [ 8988.209626] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8988.220588] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8988.223008] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 8988.225455] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 8988.228210] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 8988.230717] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 8988.233743] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9000.062474] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 9000.069784] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9000.099253] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9000.104371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9000.108061] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9000.110421] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9000.120109] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9000.123271] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9000.126169] RDX: 0000000000004ccf RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9000.129492] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9000.132848] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9000.135749] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9000.138703] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9000.142337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9000.146743] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9000.150340] Call Trace: [ 9000.151374] [ 9000.152423] ? show_regs.cold.9+0x22/0x2f [ 9000.154252] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9000.156418] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9000.158428] ? watchdog+0x40/0x40 [ 9000.159969] ? hrtimer_interrupt+0x128/0x2c0 [ 9000.161856] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9000.164097] ? apic_timer_interrupt+0xf/0x20 [ 9000.166147] [ 9000.167382] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9000.170094] do_raw_spin_lock+0x177/0x190 [ 9000.172259] _raw_spin_lock+0x21/0x30 [ 9000.174019] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9000.176565] ? do_raw_spin_unlock+0x75/0x190 [ 9000.178953] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9000.181332] ? do_wait_intr+0xf0/0xf0 [ 9000.182831] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9000.185494] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9000.188050] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9000.190810] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9000.192967] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9000.194978] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9000.197043] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9000.199147] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9000.201355] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9000.204715] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9000.207252] lmv_intent_open+0x338/0x1150 [lmv] [ 9000.209387] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9000.211821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9000.214352] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9000.216105] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9000.218779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9000.221432] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9000.223952] ? do_raw_spin_unlock+0x75/0x190 [ 9000.225826] ? _raw_spin_unlock+0x12/0x30 [ 9000.227563] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9000.230309] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9000.232765] ll_atomic_open+0x36a/0xd70 [lustre] [ 9000.235735] lookup_open+0x356/0xb40 [ 9000.237190] do_last+0x313/0xfc0 [ 9000.238407] ? nd_jump_root+0xe5/0x160 [ 9000.240202] ? path_init+0x437/0x520 [ 9000.241846] path_openat+0xf7/0x500 [ 9000.245284] ? do_raw_spin_unlock+0x75/0x190 [ 9000.248036] do_filp_open+0x99/0x140 [ 9000.250257] ? getname_flags+0x6e/0x330 [ 9000.252988] ? __check_object_size+0xff/0x256 [ 9000.256460] ? do_raw_spin_unlock+0x75/0x190 [ 9000.261784] ? _raw_spin_unlock+0x12/0x30 [ 9000.264992] do_sys_openat2+0x2b4/0x410 [ 9000.267427] do_sys_open+0x73/0xa0 [ 9000.270312] __x64_sys_openat+0x24/0x30 [ 9000.272239] do_syscall_64+0xc1/0x440 [ 9000.276249] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9000.279781] RIP: 0033:0x7fdc0afbf332 [ 9000.281877] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9000.296343] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9000.303373] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9000.308781] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9000.315960] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9000.320094] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9000.328047] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9004.048550] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 9004.053245] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 9004.054644] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9004.054909] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9004.054917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9004.054920] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9004.054936] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 9004.054940] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9004.054946] RAX: 0000000000000e06 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9004.054948] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9004.054951] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9004.054953] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9004.054955] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9004.054959] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9004.054962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9004.054965] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9004.054974] Call Trace: [ 9004.054980] [ 9004.054986] ? show_regs.cold.9+0x22/0x2f [ 9004.054997] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9004.055005] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9004.055010] ? watchdog+0x40/0x40 [ 9004.055014] ? hrtimer_interrupt+0x128/0x2c0 [ 9004.055018] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9004.055028] ? apic_timer_interrupt+0xf/0x20 [ 9004.055032] [ 9004.055036] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9004.055041] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9004.055045] ? dump_stack+0xf6/0x10e [ 9004.055051] do_raw_spin_lock+0x177/0x190 [ 9004.055056] _raw_spin_lock+0x21/0x30 [ 9004.055062] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9004.066175] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9004.066265] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9004.066273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9004.066280] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9004.066552] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9004.066562] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9004.066566] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9004.066570] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9004.066572] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9004.066575] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9004.066577] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9004.066580] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9004.066583] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9004.066586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9004.066588] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9004.066598] Call Trace: [ 9004.066603] [ 9004.066610] ? show_regs.cold.9+0x22/0x2f [ 9004.066619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9004.066625] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9004.066629] ? watchdog+0x40/0x40 [ 9004.066633] ? hrtimer_interrupt+0x128/0x2c0 [ 9004.066636] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9004.066642] ? apic_timer_interrupt+0xf/0x20 [ 9004.066647] [ 9004.066649] ? native_safe_halt+0x2e/0x50 [ 9004.066654] kvm_wait+0xcd/0xe0 [ 9004.066660] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9004.066668] do_raw_spin_lock+0x177/0x190 [ 9004.066672] _raw_spin_lock+0x21/0x30 [ 9004.066676] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9004.066879] process_one_work+0x2c8/0x700 [ 9004.066887] worker_thread+0x296/0x6e0 [ 9004.066891] ? rescuer_thread+0x570/0x570 [ 9004.066895] kthread+0x1d1/0x200 [ 9004.066900] ? set_kthread_struct+0x70/0x70 [ 9004.066903] ret_from_fork+0x1f/0x30 [ 9004.468877] obd_mod_free+0x46/0x140 [ptlrpc] [ 9004.471745] mdc_commit_open+0x91/0xb0 [mdc] [ 9004.474704] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9004.481903] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9004.485830] after_reply+0xe2c/0x1780 [ptlrpc] [ 9004.491972] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9004.494887] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9004.496971] ? do_wait_intr+0xf0/0xf0 [ 9004.501547] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9004.510692] kthread+0x1d1/0x200 [ 9004.511871] ? set_kthread_struct+0x70/0x70 [ 9004.519433] ret_from_fork+0x1f/0x30 [ 9009.634918] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6984s! [ 9009.641318] Showing busy workqueues and worker pools: [ 9009.643551] workqueue events: flags=0x0 [ 9009.645356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9009.648575] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9009.651902] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9009.655403] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9009.660647] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9009.663831] pending: kfree_rcu_monitor [ 9009.665616] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9009.668870] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9009.674624] workqueue events_long: flags=0x0 [ 9009.676997] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9009.683473] pending: lu_objects_flush [obdclass] [ 9009.686984] workqueue events_unbound: flags=0x2 [ 9009.690208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9009.692459] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9009.703783] workqueue events_power_efficient: flags=0x80 [ 9009.719191] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9009.726310] pending: neigh_periodic_work, check_lifetime [ 9009.737992] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9009.747771] pending: do_cache_clean, neigh_periodic_work [ 9009.749890] workqueue mm_percpu_wq: flags=0x8 [ 9009.758938] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9009.763220] pending: lru_add_drain_per_cpu [ 9009.767920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9009.772363] pending: vmstat_update, lru_add_drain_per_cpu [ 9009.778650] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9009.786907] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9009.792627] workqueue writeback: flags=0x4a [ 9009.795747] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9009.799655] pending: wb_workfn, wb_workfn [ 9009.804524] workqueue xprtiod: flags=0xa [ 9009.807179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9009.813342] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9009.819251] workqueue ipv6_addrconf: flags=0x40008 [ 9009.821927] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9009.825745] pending: addrconf_verify_work [ 9009.830615] workqueue knbd0-recv: flags=0x1a [ 9009.832857] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9009.837159] in-flight: 655:recv_work [ 9009.839652] workqueue ptlrpc_pinger: flags=0x2 [ 9009.842112] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9009.847178] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9009.852337] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6984s workers=4 idle: 36396 38 11188 [ 9009.860900] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9000s workers=2 idle: 658 [ 9016.054065] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 9016.062588] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9016.082967] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9016.088509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9016.092967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9016.097706] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9016.107794] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9016.111791] RAX: 0000000000005a19 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9016.115939] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9016.119485] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9016.122952] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9016.126037] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9016.129174] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9016.133075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9016.135754] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9016.139181] Call Trace: [ 9016.140417] [ 9016.141829] ? show_regs.cold.9+0x22/0x2f [ 9016.143661] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9016.145198] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9016.147783] ? watchdog+0x40/0x40 [ 9016.149111] ? hrtimer_interrupt+0x128/0x2c0 [ 9016.151119] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9016.153137] ? apic_timer_interrupt+0xf/0x20 [ 9016.155120] [ 9016.155926] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9016.158156] do_raw_spin_lock+0x177/0x190 [ 9016.159439] _raw_spin_lock+0x21/0x30 [ 9016.160792] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9016.162839] ? do_raw_spin_unlock+0x75/0x190 [ 9016.163921] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9016.166499] ? do_wait_intr+0xf0/0xf0 [ 9016.168056] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9016.170796] mdc_close+0x53b/0xd70 [mdc] [ 9016.172574] lmv_close+0x18d/0x490 [lmv] [ 9016.174423] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9016.176850] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9016.179009] ll_md_close+0x274/0x800 [lustre] [ 9016.181229] ll_file_release+0x128/0xc50 [lustre] [ 9016.183664] __fput+0x10d/0x440 [ 9016.184990] ____fput+0x12/0x20 [ 9016.186927] task_work_run+0xc6/0x110 [ 9016.188702] exit_to_usermode_loop+0x1dd/0x1f0 [ 9016.190954] do_syscall_64+0x3d6/0x440 [ 9016.192983] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9016.195185] RIP: 0033:0x7fdc0afbfcd8 [ 9016.197224] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9016.206137] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9016.209986] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9016.213787] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9016.217576] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9016.221419] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9016.227368] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9028.054870] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 9028.063022] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9028.112439] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9028.132885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9028.143264] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9028.148974] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9028.156933] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9028.160152] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9028.165083] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9028.172564] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9028.178743] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9028.192222] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9028.194570] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9028.216059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9028.221255] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9028.230689] Call Trace: [ 9028.233391] [ 9028.234196] ? show_regs.cold.9+0x22/0x2f [ 9028.236183] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9028.241338] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9028.245892] ? watchdog+0x40/0x40 [ 9028.247583] ? hrtimer_interrupt+0x128/0x2c0 [ 9028.250476] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9028.252793] ? apic_timer_interrupt+0xf/0x20 [ 9028.255163] [ 9028.255913] ? native_safe_halt+0x2e/0x50 [ 9028.258130] kvm_wait+0xcd/0xe0 [ 9028.259290] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9028.267528] do_raw_spin_lock+0x177/0x190 [ 9028.272359] _raw_spin_lock+0x21/0x30 [ 9028.274653] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9028.278425] ? do_raw_spin_unlock+0x75/0x190 [ 9028.280354] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9028.282541] ? do_wait_intr+0xf0/0xf0 [ 9028.284648] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9028.287530] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9028.290557] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9028.293815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9028.296456] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9028.299407] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9028.301606] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9028.304136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9028.306692] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9028.309710] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9028.312663] lmv_intent_open+0x338/0x1150 [lmv] [ 9028.315587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9028.320278] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9028.323572] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9028.326058] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9028.329554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9028.334184] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9028.337293] ? do_raw_spin_unlock+0x75/0x190 [ 9028.341081] ? _raw_spin_unlock+0x12/0x30 [ 9028.343934] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9028.345863] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9028.349199] ll_atomic_open+0x36a/0xd70 [lustre] [ 9028.352165] lookup_open+0x356/0xb40 [ 9028.355250] do_last+0x313/0xfc0 [ 9028.358274] ? nd_jump_root+0xe5/0x160 [ 9028.360318] ? path_init+0x437/0x520 [ 9028.364780] path_openat+0xf7/0x500 [ 9028.368671] ? do_raw_spin_unlock+0x75/0x190 [ 9028.374265] do_filp_open+0x99/0x140 [ 9028.378347] ? getname_flags+0x6e/0x330 [ 9028.380361] ? __check_object_size+0xff/0x256 [ 9028.381729] ? do_raw_spin_unlock+0x75/0x190 [ 9028.383624] ? _raw_spin_unlock+0x12/0x30 [ 9028.385720] do_sys_openat2+0x2b4/0x410 [ 9028.387739] do_sys_open+0x73/0xa0 [ 9028.389629] __x64_sys_openat+0x24/0x30 [ 9028.392148] do_syscall_64+0xc1/0x440 [ 9028.394381] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9028.397309] RIP: 0033:0x7fdc0afbf332 [ 9028.399077] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9028.409542] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9028.412630] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9028.416443] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9028.419716] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9028.424241] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9028.428309] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9032.051072] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 9032.053714] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9032.081748] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9032.085812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9032.089004] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9032.092275] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9032.096831] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9032.111196] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9032.115792] RAX: 0000000000002d0d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9032.121349] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9032.126749] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9032.130532] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9032.136916] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9032.141681] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9032.147013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9032.154992] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9032.157042] Call Trace: [ 9032.158962] [ 9032.159947] ? show_regs.cold.9+0x22/0x2f [ 9032.161153] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9032.168958] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9032.172529] ? watchdog+0x40/0x40 [ 9032.175482] ? hrtimer_interrupt+0x128/0x2c0 [ 9032.181253] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9032.185038] ? apic_timer_interrupt+0xf/0x20 [ 9032.186496] [ 9032.187136] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9032.190282] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9032.194121] do_raw_spin_lock+0x177/0x190 [ 9032.195938] _raw_spin_lock+0x21/0x30 [ 9032.199580] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9032.201406] process_one_work+0x2c8/0x700 [ 9032.202958] worker_thread+0x296/0x6e0 [ 9032.204209] ? rescuer_thread+0x570/0x570 [ 9032.209174] kthread+0x1d1/0x200 [ 9032.211210] ? set_kthread_struct+0x70/0x70 [ 9032.213658] ret_from_fork+0x1f/0x30 [ 9036.048029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 9036.051413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9036.072186] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9036.076497] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9036.079303] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9036.081102] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9036.088777] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9036.091729] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9036.094498] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9036.098176] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9036.101911] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9036.104951] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9036.107644] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9036.111354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9036.113895] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9036.117254] Call Trace: [ 9036.118333] [ 9036.119073] ? show_regs.cold.9+0x22/0x2f [ 9036.120304] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9036.122115] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9036.124027] ? watchdog+0x40/0x40 [ 9036.125028] ? hrtimer_interrupt+0x128/0x2c0 [ 9036.126523] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9036.128542] ? apic_timer_interrupt+0xf/0x20 [ 9036.131450] [ 9036.132713] ? native_safe_halt+0x2e/0x50 [ 9036.134599] kvm_wait+0xcd/0xe0 [ 9036.136106] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9036.138743] ? dump_stack+0xf6/0x10e [ 9036.140234] do_raw_spin_lock+0x177/0x190 [ 9036.142789] _raw_spin_lock+0x21/0x30 [ 9036.144161] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9036.146373] obd_mod_free+0x46/0x140 [ptlrpc] [ 9036.148084] mdc_commit_open+0x91/0xb0 [mdc] [ 9036.150367] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9036.154523] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9036.158020] after_reply+0xe2c/0x1780 [ptlrpc] [ 9036.161340] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9036.165084] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9036.166830] ? do_wait_intr+0xf0/0xf0 [ 9036.168432] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9036.170531] kthread+0x1d1/0x200 [ 9036.172196] ? set_kthread_struct+0x70/0x70 [ 9036.173983] ret_from_fork+0x1f/0x30 [ 9040.351253] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7015s! [ 9040.356914] Showing busy workqueues and worker pools: [ 9040.359440] workqueue events: flags=0x0 [ 9040.364055] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9040.369955] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9040.375734] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9040.378905] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9040.384341] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9040.387910] pending: kfree_rcu_monitor [ 9040.390044] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9040.393289] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9040.401725] workqueue events_long: flags=0x0 [ 9040.403524] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9040.408074] pending: lu_objects_flush [obdclass] [ 9040.410485] workqueue events_unbound: flags=0x2 [ 9040.412170] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9040.415039] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9040.420133] workqueue events_power_efficient: flags=0x80 [ 9040.422351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9040.425220] pending: neigh_periodic_work, check_lifetime [ 9040.427436] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9040.429954] pending: do_cache_clean, neigh_periodic_work [ 9040.432154] workqueue mm_percpu_wq: flags=0x8 [ 9040.433750] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9040.436567] pending: lru_add_drain_per_cpu [ 9040.438858] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9040.442018] pending: vmstat_update, lru_add_drain_per_cpu [ 9040.444706] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9040.447829] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9040.451051] workqueue writeback: flags=0x4a [ 9040.453338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9040.456196] pending: wb_workfn, wb_workfn [ 9040.457818] workqueue xprtiod: flags=0xa [ 9040.459453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9040.462312] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9040.465550] workqueue ipv6_addrconf: flags=0x40008 [ 9040.470754] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9040.476220] pending: addrconf_verify_work [ 9040.478337] workqueue knbd0-recv: flags=0x1a [ 9040.483597] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9040.489832] in-flight: 655:recv_work [ 9040.493065] workqueue ptlrpc_pinger: flags=0x2 [ 9040.496818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9040.499430] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9040.508944] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7015s workers=4 idle: 36396 38 11188 [ 9040.512867] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9031s workers=2 idle: 658 [ 9044.054067] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 9044.061632] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9044.114449] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9044.129619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9044.139873] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9044.148209] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9044.168141] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9044.185441] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9044.203557] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9044.208487] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9044.215777] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9044.221476] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9044.228993] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9044.239906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9044.242730] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9044.248268] Call Trace: [ 9044.249093] [ 9044.249775] ? show_regs.cold.9+0x22/0x2f [ 9044.266292] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9044.267813] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9044.273997] ? watchdog+0x40/0x40 [ 9044.275093] ? hrtimer_interrupt+0x128/0x2c0 [ 9044.276386] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9044.279919] ? apic_timer_interrupt+0xf/0x20 [ 9044.283892] [ 9044.285257] ? native_safe_halt+0x2e/0x50 [ 9044.286671] kvm_wait+0xcd/0xe0 [ 9044.288980] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9044.290796] do_raw_spin_lock+0x177/0x190 [ 9044.293562] _raw_spin_lock+0x21/0x30 [ 9044.295983] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9044.306781] ? do_raw_spin_unlock+0x75/0x190 [ 9044.312256] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9044.324319] ? do_wait_intr+0xf0/0xf0 [ 9044.326473] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9044.328500] mdc_close+0x53b/0xd70 [mdc] [ 9044.335496] lmv_close+0x18d/0x490 [lmv] [ 9044.342540] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9044.349780] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9044.357383] ll_md_close+0x274/0x800 [lustre] [ 9044.358884] ll_file_release+0x128/0xc50 [lustre] [ 9044.363084] __fput+0x10d/0x440 [ 9044.367711] ____fput+0x12/0x20 [ 9044.369880] task_work_run+0xc6/0x110 [ 9044.375006] exit_to_usermode_loop+0x1dd/0x1f0 [ 9044.380494] do_syscall_64+0x3d6/0x440 [ 9044.382345] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9044.387358] RIP: 0033:0x7fdc0afbfcd8 [ 9044.388967] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9044.402988] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9044.406120] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9044.410453] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9044.414033] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9044.419513] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9044.421815] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9056.052117] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 9056.060528] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9056.086870] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9056.092184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9056.097152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9056.099783] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9056.107766] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9056.111329] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9056.114508] RDX: 0000000000003de8 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9056.117012] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9056.120147] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9056.123921] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9056.126718] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9056.130377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9056.133224] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9056.136126] Call Trace: [ 9056.137140] [ 9056.138181] ? show_regs.cold.9+0x22/0x2f [ 9056.139921] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9056.141886] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9056.144198] ? watchdog+0x40/0x40 [ 9056.146211] ? hrtimer_interrupt+0x128/0x2c0 [ 9056.148348] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9056.151235] ? apic_timer_interrupt+0xf/0x20 [ 9056.152891] [ 9056.153520] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9056.154976] do_raw_spin_lock+0x177/0x190 [ 9056.156289] _raw_spin_lock+0x21/0x30 [ 9056.157750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9056.159816] ? do_raw_spin_unlock+0x75/0x190 [ 9056.161343] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9056.163264] ? do_wait_intr+0xf0/0xf0 [ 9056.164694] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9056.166872] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9056.169123] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9056.171309] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9056.173546] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9056.175555] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9056.177124] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9056.178277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9056.180344] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9056.182729] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9056.185023] lmv_intent_open+0x338/0x1150 [lmv] [ 9056.186641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9056.188522] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9056.190290] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9056.191875] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9056.195846] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9056.198385] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9056.201870] ? do_raw_spin_unlock+0x75/0x190 [ 9056.203848] ? _raw_spin_unlock+0x12/0x30 [ 9056.205784] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9056.208202] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9056.213574] ll_atomic_open+0x36a/0xd70 [lustre] [ 9056.216766] lookup_open+0x356/0xb40 [ 9056.218455] do_last+0x313/0xfc0 [ 9056.220205] ? nd_jump_root+0xe5/0x160 [ 9056.222422] ? path_init+0x437/0x520 [ 9056.223985] path_openat+0xf7/0x500 [ 9056.225508] ? do_raw_spin_unlock+0x75/0x190 [ 9056.227407] do_filp_open+0x99/0x140 [ 9056.228759] ? getname_flags+0x6e/0x330 [ 9056.230540] ? __check_object_size+0xff/0x256 [ 9056.232233] ? do_raw_spin_unlock+0x75/0x190 [ 9056.235259] ? _raw_spin_unlock+0x12/0x30 [ 9056.237276] do_sys_openat2+0x2b4/0x410 [ 9056.239260] do_sys_open+0x73/0xa0 [ 9056.240631] __x64_sys_openat+0x24/0x30 [ 9056.242187] do_syscall_64+0xc1/0x440 [ 9056.243657] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9056.245852] RIP: 0033:0x7fdc0afbf332 [ 9056.247412] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9056.256173] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9056.260599] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9056.265157] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9056.268613] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9056.272103] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9056.275012] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9060.053529] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 9060.058634] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9060.088335] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9060.101195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9060.111454] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9060.116731] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9060.120583] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9060.133654] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9060.137486] RAX: 0000000000007f85 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9060.140147] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9060.145616] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9060.152196] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9060.154599] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9060.164366] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9060.169219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9060.176145] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9060.186567] Call Trace: [ 9060.187534] [ 9060.188194] ? show_regs.cold.9+0x22/0x2f [ 9060.190508] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9060.191947] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9060.193689] ? watchdog+0x40/0x40 [ 9060.194968] ? hrtimer_interrupt+0x128/0x2c0 [ 9060.196590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9060.199215] ? apic_timer_interrupt+0xf/0x20 [ 9060.200725] [ 9060.201764] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9060.203686] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9060.206952] do_raw_spin_lock+0x177/0x190 [ 9060.211283] _raw_spin_lock+0x21/0x30 [ 9060.213355] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9060.215134] process_one_work+0x2c8/0x700 [ 9060.221354] worker_thread+0x296/0x6e0 [ 9060.222662] ? rescuer_thread+0x570/0x570 [ 9060.227163] kthread+0x1d1/0x200 [ 9060.233258] ? set_kthread_struct+0x70/0x70 [ 9060.234644] ret_from_fork+0x1f/0x30 [ 9068.048014] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 9068.056445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9068.092944] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9068.103632] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9068.114561] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9068.123638] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9068.143749] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9068.156446] RAX: 0000000000000f27 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9068.167462] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9068.176253] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9068.182766] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9068.186086] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9068.193408] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9068.197185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9068.207867] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9068.219322] Call Trace: [ 9068.220673] [ 9068.221349] ? show_regs.cold.9+0x22/0x2f [ 9068.223690] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9068.225905] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9068.227479] ? watchdog+0x40/0x40 [ 9068.228649] ? hrtimer_interrupt+0x128/0x2c0 [ 9068.230859] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9068.232873] ? apic_timer_interrupt+0xf/0x20 [ 9068.234481] [ 9068.235653] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9068.237929] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9068.240197] ? dump_stack+0xf6/0x10e [ 9068.242163] do_raw_spin_lock+0x177/0x190 [ 9068.243858] _raw_spin_lock+0x21/0x30 [ 9068.245805] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9068.249607] obd_mod_free+0x46/0x140 [ptlrpc] [ 9068.252194] mdc_commit_open+0x91/0xb0 [mdc] [ 9068.257659] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9068.260595] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9068.263494] after_reply+0xe2c/0x1780 [ptlrpc] [ 9068.266797] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9068.269534] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9068.272291] ? do_wait_intr+0xf0/0xf0 [ 9068.273704] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9068.276385] kthread+0x1d1/0x200 [ 9068.277624] ? set_kthread_struct+0x70/0x70 [ 9068.280083] ret_from_fork+0x1f/0x30 [ 9070.559050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7045s! [ 9070.562499] Showing busy workqueues and worker pools: [ 9070.568407] workqueue events: flags=0x0 [ 9070.570780] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9070.577792] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9070.585432] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9070.592438] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9070.599893] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9070.609095] pending: kfree_rcu_monitor [ 9070.613284] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9070.621033] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9070.632033] workqueue events_long: flags=0x0 [ 9070.636361] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9070.642205] pending: lu_objects_flush [obdclass] [ 9070.646539] workqueue events_unbound: flags=0x2 [ 9070.650429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9070.654576] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9070.661646] workqueue events_power_efficient: flags=0x80 [ 9070.666124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9070.671869] pending: neigh_periodic_work, check_lifetime [ 9070.676040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9070.681708] pending: do_cache_clean, neigh_periodic_work [ 9070.686536] workqueue mm_percpu_wq: flags=0x8 [ 9070.690178] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9070.694582] pending: lru_add_drain_per_cpu [ 9070.696550] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9070.699268] pending: vmstat_update, lru_add_drain_per_cpu [ 9070.701766] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9070.704595] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9070.707236] workqueue writeback: flags=0x4a [ 9070.713445] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9070.719321] pending: wb_workfn, wb_workfn [ 9070.720792] workqueue xprtiod: flags=0xa [ 9070.723188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9070.724803] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9070.729908] workqueue ipv6_addrconf: flags=0x40008 [ 9070.731499] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9070.733748] pending: addrconf_verify_work [ 9070.736024] workqueue knbd0-recv: flags=0x1a [ 9070.737365] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9070.741182] in-flight: 655:recv_work [ 9070.742995] workqueue ptlrpc_pinger: flags=0x2 [ 9070.744523] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9070.746861] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9070.749049] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7045s workers=4 idle: 36396 38 11188 [ 9070.751899] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9061s workers=2 idle: 658 [ 9072.054084] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 9072.059523] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9072.085547] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9072.092543] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9072.098265] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9072.103215] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9072.116908] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9072.120882] RAX: 0000000000000d07 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9072.126939] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9072.131868] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9072.136126] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9072.140533] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9072.144931] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9072.151267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9072.154247] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9072.158547] Call Trace: [ 9072.160820] [ 9072.161763] ? show_regs.cold.9+0x22/0x2f [ 9072.164704] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9072.167274] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9072.169180] ? watchdog+0x40/0x40 [ 9072.171045] ? hrtimer_interrupt+0x128/0x2c0 [ 9072.173140] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9072.175865] ? apic_timer_interrupt+0xf/0x20 [ 9072.180883] [ 9072.181846] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9072.183617] do_raw_spin_lock+0x177/0x190 [ 9072.185752] _raw_spin_lock+0x21/0x30 [ 9072.188187] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9072.193269] ? do_raw_spin_unlock+0x75/0x190 [ 9072.195791] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9072.198136] ? do_wait_intr+0xf0/0xf0 [ 9072.200720] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9072.203831] mdc_close+0x53b/0xd70 [mdc] [ 9072.206141] lmv_close+0x18d/0x490 [lmv] [ 9072.208120] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9072.211025] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9072.215889] ll_md_close+0x274/0x800 [lustre] [ 9072.218043] ll_file_release+0x128/0xc50 [lustre] [ 9072.220810] __fput+0x10d/0x440 [ 9072.222970] ____fput+0x12/0x20 [ 9072.224111] task_work_run+0xc6/0x110 [ 9072.225954] exit_to_usermode_loop+0x1dd/0x1f0 [ 9072.228377] do_syscall_64+0x3d6/0x440 [ 9072.229704] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9072.235151] RIP: 0033:0x7fdc0afbfcd8 [ 9072.239711] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9072.251554] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9072.256928] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9072.261745] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9072.269145] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9072.280537] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9072.293456] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9084.052069] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 9084.054956] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9084.074494] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9084.079678] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9084.083818] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9084.085889] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9084.094302] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9084.097374] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9084.100574] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9084.103955] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9084.107266] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9084.110256] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9084.113155] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9084.116316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9084.119406] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9084.122257] Call Trace: [ 9084.123228] [ 9084.124224] ? show_regs.cold.9+0x22/0x2f [ 9084.125923] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9084.128431] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9084.130440] ? watchdog+0x40/0x40 [ 9084.131714] ? hrtimer_interrupt+0x128/0x2c0 [ 9084.133603] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9084.134937] ? apic_timer_interrupt+0xf/0x20 [ 9084.136211] [ 9084.137098] ? native_safe_halt+0x2e/0x50 [ 9084.138837] kvm_wait+0xcd/0xe0 [ 9084.140285] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9084.141933] do_raw_spin_lock+0x177/0x190 [ 9084.143512] _raw_spin_lock+0x21/0x30 [ 9084.144975] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9084.147410] ? do_raw_spin_unlock+0x75/0x190 [ 9084.149184] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9084.151453] ? do_wait_intr+0xf0/0xf0 [ 9084.152793] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9084.154970] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9084.156954] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9084.159865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9084.162358] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9084.165762] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9084.167902] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9084.170529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9084.173699] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9084.176403] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9084.180304] lmv_intent_open+0x338/0x1150 [lmv] [ 9084.182391] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9084.184875] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9084.187303] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9084.189218] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9084.191923] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9084.195932] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9084.198751] ? do_raw_spin_unlock+0x75/0x190 [ 9084.200304] ? _raw_spin_unlock+0x12/0x30 [ 9084.204559] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9084.206976] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9084.209145] ll_atomic_open+0x36a/0xd70 [lustre] [ 9084.211165] lookup_open+0x356/0xb40 [ 9084.212404] do_last+0x313/0xfc0 [ 9084.213619] ? nd_jump_root+0xe5/0x160 [ 9084.215314] ? path_init+0x437/0x520 [ 9084.218551] path_openat+0xf7/0x500 [ 9084.220181] ? do_raw_spin_unlock+0x75/0x190 [ 9084.225850] do_filp_open+0x99/0x140 [ 9084.227344] ? getname_flags+0x6e/0x330 [ 9084.229306] ? __check_object_size+0xff/0x256 [ 9084.232129] ? do_raw_spin_unlock+0x75/0x190 [ 9084.234620] ? _raw_spin_unlock+0x12/0x30 [ 9084.235956] do_sys_openat2+0x2b4/0x410 [ 9084.237728] do_sys_open+0x73/0xa0 [ 9084.238956] __x64_sys_openat+0x24/0x30 [ 9084.241296] do_syscall_64+0xc1/0x440 [ 9084.243500] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9084.246328] RIP: 0033:0x7fdc0afbf332 [ 9084.247724] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9084.258158] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9084.261154] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9084.266691] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9084.271708] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9084.277835] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9084.281159] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9088.051035] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 9088.056696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9088.096817] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9088.104597] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9088.115966] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9088.122300] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9088.129743] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9088.160368] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9088.175677] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9088.192079] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9088.203661] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9088.210633] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9088.213818] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9088.217178] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9088.221311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9088.225181] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9088.229429] Call Trace: [ 9088.233032] [ 9088.235609] ? show_regs.cold.9+0x22/0x2f [ 9088.239301] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9088.245182] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9088.251471] ? watchdog+0x40/0x40 [ 9088.260413] ? hrtimer_interrupt+0x128/0x2c0 [ 9088.262544] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9088.268306] ? apic_timer_interrupt+0xf/0x20 [ 9088.269744] [ 9088.275128] ? native_safe_halt+0x2e/0x50 [ 9088.281700] kvm_wait+0xcd/0xe0 [ 9088.287697] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9088.296993] do_raw_spin_lock+0x177/0x190 [ 9088.304481] _raw_spin_lock+0x21/0x30 [ 9088.307179] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9088.310046] process_one_work+0x2c8/0x700 [ 9088.315354] worker_thread+0x296/0x6e0 [ 9088.318268] ? rescuer_thread+0x570/0x570 [ 9088.323841] kthread+0x1d1/0x200 [ 9088.325771] ? set_kthread_struct+0x70/0x70 [ 9088.328110] ret_from_fork+0x1f/0x30 [ 9096.048026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 9096.051158] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9096.069516] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9096.073501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9096.080671] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9096.082684] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9096.090550] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9096.093307] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9096.095591] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9096.098278] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9096.100087] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9096.101808] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9096.104404] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9096.108179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9096.110472] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9096.113650] Call Trace: [ 9096.114842] [ 9096.115855] ? show_regs.cold.9+0x22/0x2f [ 9096.120906] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9096.126688] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9096.133664] ? watchdog+0x40/0x40 [ 9096.139027] ? hrtimer_interrupt+0x128/0x2c0 [ 9096.147142] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9096.152150] ? apic_timer_interrupt+0xf/0x20 [ 9096.155857] [ 9096.157390] ? native_safe_halt+0x2e/0x50 [ 9096.159761] kvm_wait+0xcd/0xe0 [ 9096.162359] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9096.165802] ? dump_stack+0xf6/0x10e [ 9096.167750] do_raw_spin_lock+0x177/0x190 [ 9096.172844] _raw_spin_lock+0x21/0x30 [ 9096.174464] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9096.180840] obd_mod_free+0x46/0x140 [ptlrpc] [ 9096.191113] mdc_commit_open+0x91/0xb0 [mdc] [ 9096.199255] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9096.203158] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9096.207933] after_reply+0xe2c/0x1780 [ptlrpc] [ 9096.210735] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9096.214124] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9096.219804] ? do_wait_intr+0xf0/0xf0 [ 9096.223538] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9096.227443] kthread+0x1d1/0x200 [ 9096.229306] ? set_kthread_struct+0x70/0x70 [ 9096.231921] ret_from_fork+0x1f/0x30 [ 9099.510033] rcu: INFO: rcu_sched self-detected stall on CPU [ 9099.512913] rcu: 0-...!: (4440138 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 9099.517061] (t=7080167 jiffies g=270761 q=4416) [ 9099.518205] rcu: rcu_sched kthread starved for 7075099 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 9099.523283] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9099.527541] rcu: RCU grace-period kthread stack dump: [ 9099.529630] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9099.533715] Call Trace: [ 9099.534979] __schedule+0x351/0xcb0 [ 9099.537454] schedule+0xc0/0x180 [ 9099.539293] schedule_timeout+0xb4/0x190 [ 9099.540917] ? do_raw_spin_unlock+0x75/0x190 [ 9099.542930] ? __next_timer_interrupt+0x160/0x160 [ 9099.545439] ? prepare_to_swait_event+0xf3/0x140 [ 9099.547435] rcu_gp_kthread+0x8dd/0x15b0 [ 9099.549242] ? force_qs_rnp+0x2c0/0x2c0 [ 9099.550647] kthread+0x1d1/0x200 [ 9099.551643] ? set_kthread_struct+0x70/0x70 [ 9099.552938] ret_from_fork+0x1f/0x30 [ 9099.554450] rcu: Stack dump where RCU GP kthread last ran: [ 9099.557052] Sending NMI from CPU 0 to CPUs 2: [ 9099.570075] NMI backtrace for cpu 2 [ 9099.570082] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9099.570086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9099.570087] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9099.570089] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9099.570092] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 [ 9099.570095] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9099.570096] RDX: 0000000000000aae RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9099.570098] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9099.570099] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9099.570100] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9099.570102] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9099.570103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9099.570104] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9099.570105] Call Trace: [ 9099.570106] [ 9099.570107] ? show_regs.cold.9+0x22/0x2f [ 9099.570108] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9099.570109] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9099.570110] ? nmi_handle+0xbd/0x210 [ 9099.570112] ? default_do_nmi+0x4b/0x1b0 [ 9099.570113] ? do_nmi+0x280/0x380 [ 9099.570114] ? end_repeat_nmi+0x16/0x1f [ 9099.570115] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9099.570116] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9099.570117] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9099.570118] [ 9099.570118] do_raw_spin_lock+0x177/0x190 [ 9099.570119] _raw_spin_lock+0x21/0x30 [ 9099.570121] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9099.570122] ? do_raw_spin_unlock+0x75/0x190 [ 9099.570123] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9099.570124] ? do_wait_intr+0xf0/0xf0 [ 9099.570125] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9099.570126] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9099.570128] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9099.570128] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9099.570130] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9099.570131] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9099.570132] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9099.570133] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9099.570134] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9099.570135] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9099.570136] lmv_intent_open+0x338/0x1150 [lmv] [ 9099.570138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9099.570139] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9099.570140] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9099.570141] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9099.570142] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9099.570143] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9099.570144] ? do_raw_spin_unlock+0x75/0x190 [ 9099.570145] ? _raw_spin_unlock+0x12/0x30 [ 9099.570146] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9099.570147] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9099.570149] ll_atomic_open+0x36a/0xd70 [lustre] [ 9099.570150] lookup_open+0x356/0xb40 [ 9099.570151] do_last+0x313/0xfc0 [ 9099.570152] ? nd_jump_root+0xe5/0x160 [ 9099.570153] ? path_init+0x437/0x520 [ 9099.570154] path_openat+0xf7/0x500 [ 9099.570156] ? do_raw_spin_unlock+0x75/0x190 [ 9099.570157] do_filp_open+0x99/0x140 [ 9099.570158] ? getname_flags+0x6e/0x330 [ 9099.570160] ? __check_object_size+0xff/0x256 [ 9099.570161] ? do_raw_spin_unlock+0x75/0x190 [ 9099.570162] ? _raw_spin_unlock+0x12/0x30 [ 9099.570163] do_sys_openat2+0x2b4/0x410 [ 9099.570164] do_sys_open+0x73/0xa0 [ 9099.570165] __x64_sys_openat+0x24/0x30 [ 9099.570166] do_syscall_64+0xc1/0x440 [ 9099.570167] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9099.570169] RIP: 0033:0x7fdc0afbf332 [ 9099.570170] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9099.570173] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9099.570176] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9099.570178] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9099.570179] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9099.570181] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9099.570183] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9099.570507] NMI backtrace for cpu 0 [ 9099.804258] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9099.813157] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9099.817049] Call Trace: [ 9099.818449] [ 9099.819589] ? dump_stack+0xbb/0x10e [ 9099.823303] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9099.827580] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9099.831057] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9099.834232] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9099.839674] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9099.844624] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9099.848692] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9099.850400] ? update_process_times+0x6d/0xc0 [ 9099.851740] ? tick_sched_handle.isra.18+0x27/0x90 [ 9099.857479] ? tick_sched_timer+0x53/0xb0 [ 9099.863358] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9099.867003] ? tick_sched_handle.isra.18+0x90/0x90 [ 9099.872360] ? hrtimer_interrupt+0x128/0x2c0 [ 9099.875064] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9099.877362] ? apic_timer_interrupt+0xf/0x20 [ 9099.880461] [ 9099.881323] ? native_safe_halt+0x2e/0x50 [ 9099.883932] ? kvm_wait+0xcd/0xe0 [ 9099.885501] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9099.887872] ? dump_stack+0xf6/0x10e [ 9099.890114] ? do_raw_spin_lock+0x177/0x190 [ 9099.892117] ? _raw_spin_lock+0x21/0x30 [ 9099.894335] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9099.897424] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9099.899555] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9099.901710] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9099.904380] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9099.906916] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9099.909829] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9099.912877] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9099.915588] ? do_wait_intr+0xf0/0xf0 [ 9099.917574] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9099.920779] ? kthread+0x1d1/0x200 [ 9099.922227] ? set_kthread_struct+0x70/0x70 [ 9099.924304] ? ret_from_fork+0x1f/0x30 [ 9100.057252] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 9100.074173] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9100.160058] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9100.176667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9100.181746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9100.185987] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9100.195819] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9100.199427] RAX: 0000000000007cb4 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9100.203968] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9100.208182] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9100.216000] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9100.223579] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9100.232864] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9100.238389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9100.240496] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9100.244600] Call Trace: [ 9100.247506] [ 9100.248596] ? show_regs.cold.9+0x22/0x2f [ 9100.250463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9100.253585] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9100.255555] ? watchdog+0x40/0x40 [ 9100.257121] ? hrtimer_interrupt+0x128/0x2c0 [ 9100.259026] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9100.260880] ? apic_timer_interrupt+0xf/0x20 [ 9100.264134] [ 9100.265166] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9100.267557] do_raw_spin_lock+0x177/0x190 [ 9100.269593] _raw_spin_lock+0x21/0x30 [ 9100.270930] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9100.273947] ? do_raw_spin_unlock+0x75/0x190 [ 9100.275416] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9100.277504] ? do_wait_intr+0xf0/0xf0 [ 9100.279015] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9100.283666] mdc_close+0x53b/0xd70 [mdc] [ 9100.285067] lmv_close+0x18d/0x490 [lmv] [ 9100.286688] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9100.289063] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9100.291433] ll_md_close+0x274/0x800 [lustre] [ 9100.295734] ll_file_release+0x128/0xc50 [lustre] [ 9100.297735] __fput+0x10d/0x440 [ 9100.299148] ____fput+0x12/0x20 [ 9100.300435] task_work_run+0xc6/0x110 [ 9100.301977] exit_to_usermode_loop+0x1dd/0x1f0 [ 9100.305993] do_syscall_64+0x3d6/0x440 [ 9100.307593] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9100.309732] RIP: 0033:0x7fdc0afbfcd8 [ 9100.311260] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9100.318941] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9100.326815] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9100.333650] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9100.338171] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9100.342182] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9100.344518] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9100.772723] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7075s! [ 9100.783973] Showing busy workqueues and worker pools: [ 9100.798821] workqueue events: flags=0x0 [ 9100.802406] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9100.816225] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9100.822450] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9100.837264] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9100.857804] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9100.870989] pending: kfree_rcu_monitor [ 9100.876605] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9100.890711] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9100.915049] workqueue events_long: flags=0x0 [ 9100.916179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9100.918404] pending: lu_objects_flush [obdclass] [ 9100.925816] workqueue events_unbound: flags=0x2 [ 9100.930931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9100.941924] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9100.960553] workqueue events_power_efficient: flags=0x80 [ 9100.967308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9100.976812] pending: neigh_periodic_work, check_lifetime [ 9100.991808] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9101.004329] pending: do_cache_clean, neigh_periodic_work [ 9101.008605] workqueue mm_percpu_wq: flags=0x8 [ 9101.013914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9101.022368] pending: lru_add_drain_per_cpu [ 9101.025128] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9101.028997] pending: vmstat_update, lru_add_drain_per_cpu [ 9101.031983] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9101.035865] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9101.039454] workqueue writeback: flags=0x4a [ 9101.042035] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9101.045702] pending: wb_workfn, wb_workfn [ 9101.047773] workqueue xprtiod: flags=0xa [ 9101.049234] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9101.052114] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9101.055759] workqueue ipv6_addrconf: flags=0x40008 [ 9101.057976] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9101.060338] pending: addrconf_verify_work [ 9101.062603] workqueue knbd0-recv: flags=0x1a [ 9101.064675] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9101.067688] in-flight: 655:recv_work [ 9101.069072] workqueue ptlrpc_pinger: flags=0x2 [ 9101.070873] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9101.073157] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9101.076223] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7076s workers=4 idle: 36396 38 11188 [ 9101.080305] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9091s workers=2 idle: 658 [ 9116.051088] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 9116.055169] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9116.093363] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9116.101104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9116.105775] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9116.109142] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9116.112999] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9116.121285] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9116.125105] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9116.128015] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9116.131973] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9116.134710] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9116.137226] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9116.141117] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9116.143784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9116.146729] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9116.150774] Call Trace: [ 9116.151641] [ 9116.152201] ? show_regs.cold.9+0x22/0x2f [ 9116.153160] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9116.154367] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9116.157045] ? watchdog+0x40/0x40 [ 9116.158204] ? hrtimer_interrupt+0x128/0x2c0 [ 9116.160697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9116.163275] ? apic_timer_interrupt+0xf/0x20 [ 9116.164694] [ 9116.166537] ? native_safe_halt+0x2e/0x50 [ 9116.167967] kvm_wait+0xcd/0xe0 [ 9116.169293] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9116.171742] do_raw_spin_lock+0x177/0x190 [ 9116.173924] _raw_spin_lock+0x21/0x30 [ 9116.175226] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9116.177061] process_one_work+0x2c8/0x700 [ 9116.181539] worker_thread+0x296/0x6e0 [ 9116.184198] ? rescuer_thread+0x570/0x570 [ 9116.186000] kthread+0x1d1/0x200 [ 9116.187089] ? set_kthread_struct+0x70/0x70 [ 9116.189013] ret_from_fork+0x1f/0x30 [ 9124.053230] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 9124.059818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9124.084896] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9124.088723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9124.091723] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9124.093725] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9124.101940] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9124.104919] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9124.107183] RDX: 0000000000002768 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9124.110070] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9124.113218] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9124.115962] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9124.118294] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9124.121176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9124.123595] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9124.126016] Call Trace: [ 9124.127078] [ 9124.128504] ? show_regs.cold.9+0x22/0x2f [ 9124.130240] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9124.132966] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9124.134890] ? watchdog+0x40/0x40 [ 9124.136396] ? hrtimer_interrupt+0x128/0x2c0 [ 9124.139446] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9124.143780] ? apic_timer_interrupt+0xf/0x20 [ 9124.148618] [ 9124.150092] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9124.155188] do_raw_spin_lock+0x177/0x190 [ 9124.157685] _raw_spin_lock+0x21/0x30 [ 9124.162945] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9124.168366] ? do_raw_spin_unlock+0x75/0x190 [ 9124.171434] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9124.177322] ? do_wait_intr+0xf0/0xf0 [ 9124.180539] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9124.184659] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9124.194452] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9124.202578] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9124.209056] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9124.215663] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9124.219924] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9124.223907] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9124.231919] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9124.234749] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9124.237336] lmv_intent_open+0x338/0x1150 [lmv] [ 9124.239327] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9124.242775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9124.245469] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9124.247743] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9124.249448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9124.252152] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9124.254556] ? do_raw_spin_unlock+0x75/0x190 [ 9124.256882] ? _raw_spin_unlock+0x12/0x30 [ 9124.258492] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9124.260987] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9124.263429] ll_atomic_open+0x36a/0xd70 [lustre] [ 9124.265410] lookup_open+0x356/0xb40 [ 9124.266547] do_last+0x313/0xfc0 [ 9124.267703] ? nd_jump_root+0xe5/0x160 [ 9124.268892] ? path_init+0x437/0x520 [ 9124.270619] path_openat+0xf7/0x500 [ 9124.272173] ? do_raw_spin_unlock+0x75/0x190 [ 9124.274706] do_filp_open+0x99/0x140 [ 9124.276072] ? getname_flags+0x6e/0x330 [ 9124.278614] ? __check_object_size+0xff/0x256 [ 9124.280631] ? do_raw_spin_unlock+0x75/0x190 [ 9124.282917] ? _raw_spin_unlock+0x12/0x30 [ 9124.284377] do_sys_openat2+0x2b4/0x410 [ 9124.286338] do_sys_open+0x73/0xa0 [ 9124.288223] __x64_sys_openat+0x24/0x30 [ 9124.290347] do_syscall_64+0xc1/0x440 [ 9124.291733] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9124.293688] RIP: 0033:0x7fdc0afbf332 [ 9124.295680] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9124.305150] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9124.309367] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9124.313016] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9124.315956] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9124.319874] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9124.322795] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9128.048518] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 9128.054115] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 9128.056512] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9128.059171] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9128.059248] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9128.059255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9128.059258] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9128.060012] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9128.060016] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9128.060021] RAX: 0000000000004f81 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9128.060023] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9128.060025] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9128.060027] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9128.060030] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9128.060033] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9128.060036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9128.060038] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9128.060045] Call Trace: [ 9128.060050] [ 9128.060065] ? show_regs.cold.9+0x22/0x2f [ 9128.060074] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9128.060079] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9128.060083] ? watchdog+0x40/0x40 [ 9128.060087] ? hrtimer_interrupt+0x128/0x2c0 [ 9128.060091] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9128.060096] ? apic_timer_interrupt+0xf/0x20 [ 9128.060100] [ 9128.060103] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9128.060109] do_raw_spin_lock+0x177/0x190 [ 9128.060113] _raw_spin_lock+0x21/0x30 [ 9128.060117] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9128.060488] ? do_raw_spin_unlock+0x75/0x190 [ 9128.060494] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9128.060834] ? do_wait_intr+0xf0/0xf0 [ 9128.060843] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9128.061030] mdc_close+0x53b/0xd70 [mdc] [ 9128.061184] lmv_close+0x18d/0x490 [lmv] [ 9128.061212] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9128.061447] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9128.061529] ll_md_close+0x274/0x800 [lustre] [ 9128.061609] ll_file_release+0x128/0xc50 [lustre] [ 9128.061764] __fput+0x10d/0x440 [ 9128.061774] ____fput+0x12/0x20 [ 9128.061778] task_work_run+0xc6/0x110 [ 9128.061784] exit_to_usermode_loop+0x1dd/0x1f0 [ 9128.061790] do_syscall_64+0x3d6/0x440 [ 9128.061794] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9128.061802] RIP: 0033:0x7fdc0afbfcd8 [ 9128.061808] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9128.061812] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9128.061816] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9128.061818] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9128.061820] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9128.061822] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9128.061824] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9128.308430] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9128.313535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9128.320700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9128.323798] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9128.337605] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9128.342075] RAX: 00000000000061e2 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9128.345970] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9128.352887] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9128.360530] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9128.364241] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9128.367608] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9128.372233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9128.375630] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9128.378863] Call Trace: [ 9128.379931] [ 9128.380901] ? show_regs.cold.9+0x22/0x2f [ 9128.382939] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9128.385557] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9128.388040] ? watchdog+0x40/0x40 [ 9128.390005] ? hrtimer_interrupt+0x128/0x2c0 [ 9128.392396] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9128.394500] ? apic_timer_interrupt+0xf/0x20 [ 9128.396569] [ 9128.397570] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9128.400859] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9128.403896] ? dump_stack+0xf6/0x10e [ 9128.405556] do_raw_spin_lock+0x177/0x190 [ 9128.407475] _raw_spin_lock+0x21/0x30 [ 9128.411518] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9128.419510] obd_mod_free+0x46/0x140 [ptlrpc] [ 9128.426430] mdc_commit_open+0x91/0xb0 [mdc] [ 9128.428717] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9128.434548] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9128.442027] after_reply+0xe2c/0x1780 [ptlrpc] [ 9128.447542] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9128.453891] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9128.456517] ? do_wait_intr+0xf0/0xf0 [ 9128.458379] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9128.463268] kthread+0x1d1/0x200 [ 9128.464500] ? set_kthread_struct+0x70/0x70 [ 9128.467234] ret_from_fork+0x1f/0x30 [ 9131.487032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7106s! [ 9131.491362] Showing busy workqueues and worker pools: [ 9131.493934] workqueue events: flags=0x0 [ 9131.495953] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9131.499503] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9131.503223] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9131.506658] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9131.511807] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9131.515025] pending: kfree_rcu_monitor [ 9131.516973] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9131.520566] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9131.527209] workqueue events_long: flags=0x0 [ 9131.529497] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9131.533152] pending: lu_objects_flush [obdclass] [ 9131.537553] workqueue events_unbound: flags=0x2 [ 9131.541982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9131.550824] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9131.564182] workqueue events_power_efficient: flags=0x80 [ 9131.572981] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9131.578419] pending: neigh_periodic_work, check_lifetime [ 9131.582256] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9131.589488] pending: do_cache_clean, neigh_periodic_work [ 9131.595927] workqueue mm_percpu_wq: flags=0x8 [ 9131.600776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9131.604472] pending: lru_add_drain_per_cpu [ 9131.606821] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9131.610094] pending: vmstat_update, lru_add_drain_per_cpu [ 9131.612705] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9131.615031] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9131.617890] workqueue writeback: flags=0x4a [ 9131.619365] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9131.621250] pending: wb_workfn, wb_workfn [ 9131.623399] workqueue xprtiod: flags=0xa [ 9131.625215] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9131.627826] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9131.630946] workqueue ipv6_addrconf: flags=0x40008 [ 9131.632943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9131.635405] pending: addrconf_verify_work [ 9131.637187] workqueue knbd0-recv: flags=0x1a [ 9131.639183] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9131.642488] in-flight: 655:recv_work [ 9131.644141] workqueue ptlrpc_pinger: flags=0x2 [ 9131.646352] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9131.648878] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9131.652756] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7106s workers=4 idle: 36396 38 11188 [ 9131.662755] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9122s workers=2 idle: 658 [ 9144.051649] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 9144.068377] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9144.108460] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9144.118830] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9144.125234] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9144.132641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9144.138882] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9144.155264] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9144.158935] RAX: 0000000000003287 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9144.169660] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9144.175508] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9144.177764] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9144.183453] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9144.199167] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9144.203318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9144.205090] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9144.210847] Call Trace: [ 9144.211971] [ 9144.212683] ? show_regs.cold.9+0x22/0x2f [ 9144.216412] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9144.218629] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9144.220388] ? watchdog+0x40/0x40 [ 9144.221402] ? hrtimer_interrupt+0x128/0x2c0 [ 9144.231403] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9144.233945] ? apic_timer_interrupt+0xf/0x20 [ 9144.235544] [ 9144.237054] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9144.239311] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9144.242242] do_raw_spin_lock+0x177/0x190 [ 9144.247903] _raw_spin_lock+0x21/0x30 [ 9144.249935] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9144.252515] process_one_work+0x2c8/0x700 [ 9144.258014] worker_thread+0x296/0x6e0 [ 9144.260354] ? rescuer_thread+0x570/0x570 [ 9144.266564] kthread+0x1d1/0x200 [ 9144.268317] ? set_kthread_struct+0x70/0x70 [ 9144.271140] ret_from_fork+0x1f/0x30 [ 9152.053344] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 9152.060521] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9152.098837] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9152.105823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9152.112134] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9152.116207] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9152.130050] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9152.135806] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9152.140216] RDX: 00000000000065f1 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9152.145754] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9152.150897] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9152.157079] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9152.160850] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9152.170779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9152.176141] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9152.181621] Call Trace: [ 9152.182549] [ 9152.183227] ? show_regs.cold.9+0x22/0x2f [ 9152.186733] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9152.192194] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9152.200545] ? watchdog+0x40/0x40 [ 9152.202983] ? hrtimer_interrupt+0x128/0x2c0 [ 9152.206885] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9152.211817] ? apic_timer_interrupt+0xf/0x20 [ 9152.213850] [ 9152.214854] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9152.216687] do_raw_spin_lock+0x177/0x190 [ 9152.218909] _raw_spin_lock+0x21/0x30 [ 9152.222670] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9152.227114] ? do_raw_spin_unlock+0x75/0x190 [ 9152.230956] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9152.235918] ? do_wait_intr+0xf0/0xf0 [ 9152.238793] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9152.242949] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9152.248609] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9152.253717] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9152.257941] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9152.263573] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9152.266753] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9152.270611] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9152.275508] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9152.279443] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9152.281778] lmv_intent_open+0x338/0x1150 [lmv] [ 9152.287007] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9152.291215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9152.303722] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9152.305308] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9152.315097] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9152.320175] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9152.323512] ? do_raw_spin_unlock+0x75/0x190 [ 9152.327145] ? _raw_spin_unlock+0x12/0x30 [ 9152.329417] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9152.331555] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9152.333675] ll_atomic_open+0x36a/0xd70 [lustre] [ 9152.335900] lookup_open+0x356/0xb40 [ 9152.337450] do_last+0x313/0xfc0 [ 9152.338810] ? nd_jump_root+0xe5/0x160 [ 9152.340165] ? path_init+0x437/0x520 [ 9152.341775] path_openat+0xf7/0x500 [ 9152.343175] ? do_raw_spin_unlock+0x75/0x190 [ 9152.345001] do_filp_open+0x99/0x140 [ 9152.346869] ? getname_flags+0x6e/0x330 [ 9152.348293] ? __check_object_size+0xff/0x256 [ 9152.355850] ? do_raw_spin_unlock+0x75/0x190 [ 9152.357231] ? _raw_spin_unlock+0x12/0x30 [ 9152.360216] do_sys_openat2+0x2b4/0x410 [ 9152.364019] do_sys_open+0x73/0xa0 [ 9152.368970] __x64_sys_openat+0x24/0x30 [ 9152.370210] do_syscall_64+0xc1/0x440 [ 9152.376331] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9152.378371] RIP: 0033:0x7fdc0afbf332 [ 9152.383501] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9152.397721] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9152.404093] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9152.411053] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9152.417960] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9152.424732] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9152.428172] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9156.048098] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 9156.050806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9156.056620] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 9156.092598] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9156.092612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9156.092614] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9156.092627] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9156.092631] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9156.092637] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9156.092639] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9156.092641] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9156.092643] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9156.092645] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9156.092648] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9156.092651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9156.092653] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9156.092661] Call Trace: [ 9156.092666] [ 9156.092672] ? show_regs.cold.9+0x22/0x2f [ 9156.092681] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9156.092687] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9156.092692] ? watchdog+0x40/0x40 [ 9156.092695] ? hrtimer_interrupt+0x128/0x2c0 [ 9156.092699] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9156.092705] ? apic_timer_interrupt+0xf/0x20 [ 9156.092709] [ 9156.092711] ? native_safe_halt+0x2e/0x50 [ 9156.092714] kvm_wait+0xcd/0xe0 [ 9156.092720] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9156.092726] ? dump_stack+0xf6/0x10e [ 9156.092731] do_raw_spin_lock+0x177/0x190 [ 9156.092735] _raw_spin_lock+0x21/0x30 [ 9156.092741] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9156.095092] obd_mod_free+0x46/0x140 [ptlrpc] [ 9156.101990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9156.102085] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9156.102092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9156.102095] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9156.102111] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9156.102116] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9156.102120] RAX: 0000000000007f84 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9156.102407] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9156.102409] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9156.102411] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9156.102413] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9156.102417] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9156.102419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9156.102422] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9156.102430] Call Trace: [ 9156.102435] [ 9156.102440] ? show_regs.cold.9+0x22/0x2f [ 9156.102449] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9156.102455] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9156.102460] ? watchdog+0x40/0x40 [ 9156.102464] ? hrtimer_interrupt+0x128/0x2c0 [ 9156.102468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9156.102477] ? apic_timer_interrupt+0xf/0x20 [ 9156.102488] [ 9156.102492] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9156.102499] do_raw_spin_lock+0x177/0x190 [ 9156.102508] _raw_spin_lock+0x21/0x30 [ 9156.102513] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9156.119603] mdc_commit_open+0x91/0xb0 [mdc] [ 9156.119658] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9156.119938] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9156.120465] after_reply+0xe2c/0x1780 [ptlrpc] [ 9156.123015] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9156.123488] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9156.123705] ? do_wait_intr+0xf0/0xf0 [ 9156.123716] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9156.124630] kthread+0x1d1/0x200 [ 9156.124637] ? set_kthread_struct+0x70/0x70 [ 9156.124641] ret_from_fork+0x1f/0x30 [ 9156.492071] ? do_raw_spin_unlock+0x75/0x190 [ 9156.494698] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9156.500838] ? do_wait_intr+0xf0/0xf0 [ 9156.505399] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9156.511132] mdc_close+0x53b/0xd70 [mdc] [ 9156.512933] lmv_close+0x18d/0x490 [lmv] [ 9156.515519] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9156.518547] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9156.521899] ll_md_close+0x274/0x800 [lustre] [ 9156.524997] ll_file_release+0x128/0xc50 [lustre] [ 9156.528078] __fput+0x10d/0x440 [ 9156.529896] ____fput+0x12/0x20 [ 9156.531538] task_work_run+0xc6/0x110 [ 9156.533782] exit_to_usermode_loop+0x1dd/0x1f0 [ 9156.536356] do_syscall_64+0x3d6/0x440 [ 9156.537835] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9156.540857] RIP: 0033:0x7fdc0afbfcd8 [ 9156.542970] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9156.553412] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9156.557513] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9156.562902] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9156.565976] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9156.571816] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9156.579427] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9161.698569] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7136s! [ 9161.706051] Showing busy workqueues and worker pools: [ 9161.710210] workqueue events: flags=0x0 [ 9161.713139] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9161.719207] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9161.725795] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9161.731191] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9161.738252] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9161.742874] pending: kfree_rcu_monitor [ 9161.745051] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9161.749915] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9161.759375] workqueue events_long: flags=0x0 [ 9161.762844] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9161.767505] pending: lu_objects_flush [obdclass] [ 9161.770789] workqueue events_unbound: flags=0x2 [ 9161.774319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9161.782045] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9161.790670] workqueue events_power_efficient: flags=0x80 [ 9161.794498] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9161.799642] pending: neigh_periodic_work, check_lifetime [ 9161.803935] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9161.809891] pending: do_cache_clean, neigh_periodic_work [ 9161.814322] workqueue mm_percpu_wq: flags=0x8 [ 9161.817461] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9161.821569] pending: lru_add_drain_per_cpu [ 9161.824269] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9161.828308] pending: vmstat_update, lru_add_drain_per_cpu [ 9161.832285] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9161.836412] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9161.840787] workqueue writeback: flags=0x4a [ 9161.844302] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9161.855502] pending: wb_workfn, wb_workfn [ 9161.861096] workqueue xprtiod: flags=0xa [ 9161.864136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9161.868984] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9161.873193] workqueue ipv6_addrconf: flags=0x40008 [ 9161.876083] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9161.880228] pending: addrconf_verify_work [ 9161.883754] workqueue knbd0-recv: flags=0x1a [ 9161.886405] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9161.890849] in-flight: 655:recv_work [ 9161.894028] workqueue ptlrpc_pinger: flags=0x2 [ 9161.897909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9161.901907] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9161.906678] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7136s workers=4 idle: 36396 38 11188 [ 9161.911673] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9152s workers=2 idle: 658 [ 9172.051024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 9172.057320] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9172.094245] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9172.099480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9172.110638] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9172.118940] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9172.121077] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9172.134656] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9172.146107] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9172.153180] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9172.160453] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9172.164773] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9172.168708] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9172.183392] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9172.199863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9172.204543] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9172.218839] Call Trace: [ 9172.222197] [ 9172.223393] ? show_regs.cold.9+0x22/0x2f [ 9172.229145] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9172.232623] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9172.240496] ? watchdog+0x40/0x40 [ 9172.246412] ? hrtimer_interrupt+0x128/0x2c0 [ 9172.255844] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9172.262317] ? apic_timer_interrupt+0xf/0x20 [ 9172.265348] [ 9172.267754] ? native_safe_halt+0x2e/0x50 [ 9172.271172] kvm_wait+0xcd/0xe0 [ 9172.273368] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9172.277354] do_raw_spin_lock+0x177/0x190 [ 9172.280192] _raw_spin_lock+0x21/0x30 [ 9172.282127] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9172.285525] process_one_work+0x2c8/0x700 [ 9172.287362] worker_thread+0x296/0x6e0 [ 9172.288514] ? rescuer_thread+0x570/0x570 [ 9172.292064] kthread+0x1d1/0x200 [ 9172.295256] ? set_kthread_struct+0x70/0x70 [ 9172.300807] ret_from_fork+0x1f/0x30 [ 9180.052493] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 9180.055126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9180.077552] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9180.082808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9180.085875] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9180.089190] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9180.096830] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9180.100267] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9180.103097] RDX: 00000000000012a7 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9180.106552] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9180.109212] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9180.111702] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9180.114663] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9180.116561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9180.119989] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9180.123920] Call Trace: [ 9180.124885] [ 9180.126001] ? show_regs.cold.9+0x22/0x2f [ 9180.127205] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9180.128601] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9180.129997] ? watchdog+0x40/0x40 [ 9180.131041] ? hrtimer_interrupt+0x128/0x2c0 [ 9180.131991] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9180.132850] ? apic_timer_interrupt+0xf/0x20 [ 9180.133917] [ 9180.134388] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9180.137825] do_raw_spin_lock+0x177/0x190 [ 9180.139025] _raw_spin_lock+0x21/0x30 [ 9180.140029] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9180.145604] ? do_raw_spin_unlock+0x75/0x190 [ 9180.146722] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9180.148284] ? do_wait_intr+0xf0/0xf0 [ 9180.150054] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9180.153028] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9180.156539] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9180.159399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9180.162309] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9180.166938] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9180.170273] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9180.171909] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9180.174523] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9180.177340] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9180.179238] lmv_intent_open+0x338/0x1150 [lmv] [ 9180.180294] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9180.187464] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9180.190125] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9180.191900] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9180.194758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9180.198001] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9180.206902] ? do_raw_spin_unlock+0x75/0x190 [ 9180.208363] ? _raw_spin_unlock+0x12/0x30 [ 9180.210333] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9180.219158] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9180.224880] ll_atomic_open+0x36a/0xd70 [lustre] [ 9180.226365] lookup_open+0x356/0xb40 [ 9180.231046] do_last+0x313/0xfc0 [ 9180.232258] ? nd_jump_root+0xe5/0x160 [ 9180.235384] ? path_init+0x437/0x520 [ 9180.238095] path_openat+0xf7/0x500 [ 9180.239382] ? do_raw_spin_unlock+0x75/0x190 [ 9180.243363] do_filp_open+0x99/0x140 [ 9180.245507] ? getname_flags+0x6e/0x330 [ 9180.248434] ? __check_object_size+0xff/0x256 [ 9180.249731] ? do_raw_spin_unlock+0x75/0x190 [ 9180.250928] ? _raw_spin_unlock+0x12/0x30 [ 9180.252103] do_sys_openat2+0x2b4/0x410 [ 9180.254426] do_sys_open+0x73/0xa0 [ 9180.256305] __x64_sys_openat+0x24/0x30 [ 9180.258131] do_syscall_64+0xc1/0x440 [ 9180.259696] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9180.263044] RIP: 0033:0x7fdc0afbf332 [ 9180.264328] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9180.273526] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9180.277283] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9180.282570] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9180.285385] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9180.288257] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9180.291600] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9184.054037] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 9184.057568] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9184.087457] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9184.091865] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9184.095244] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9184.097836] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9184.105177] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9184.108448] RAX: 0000000000005490 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9184.110967] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9184.114434] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9184.117774] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9184.121163] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9184.124726] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9184.128817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9184.132629] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9184.135633] Call Trace: [ 9184.137389] [ 9184.138261] ? show_regs.cold.9+0x22/0x2f [ 9184.140494] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9184.142562] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9184.145474] ? watchdog+0x40/0x40 [ 9184.146539] ? hrtimer_interrupt+0x128/0x2c0 [ 9184.147810] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9184.150953] ? apic_timer_interrupt+0xf/0x20 [ 9184.153571] [ 9184.155817] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9184.160459] do_raw_spin_lock+0x177/0x190 [ 9184.164978] _raw_spin_lock+0x21/0x30 [ 9184.167308] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9184.171616] ? do_raw_spin_unlock+0x75/0x190 [ 9184.173063] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9184.176600] ? do_wait_intr+0xf0/0xf0 [ 9184.178529] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9184.182782] mdc_close+0x53b/0xd70 [mdc] [ 9184.184413] lmv_close+0x18d/0x490 [lmv] [ 9184.186572] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9184.189454] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9184.190639] ll_md_close+0x274/0x800 [lustre] [ 9184.191736] ll_file_release+0x128/0xc50 [lustre] [ 9184.194225] __fput+0x10d/0x440 [ 9184.195615] ____fput+0x12/0x20 [ 9184.196897] task_work_run+0xc6/0x110 [ 9184.198484] exit_to_usermode_loop+0x1dd/0x1f0 [ 9184.200740] do_syscall_64+0x3d6/0x440 [ 9184.202598] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9184.204937] RIP: 0033:0x7fdc0afbfcd8 [ 9184.206758] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9184.216711] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9184.220656] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9184.224374] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9184.227530] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9184.234653] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9184.238174] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9188.048084] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 9188.050760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9188.089791] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9188.097416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9188.104069] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9188.109897] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9188.128907] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9188.136580] RAX: 0000000000007021 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9188.141074] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9188.145826] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9188.148938] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9188.153293] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9188.155482] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9188.158149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9188.160342] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9188.162545] Call Trace: [ 9188.163156] [ 9188.163801] ? show_regs.cold.9+0x22/0x2f [ 9188.165112] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9188.168877] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9188.171047] ? watchdog+0x40/0x40 [ 9188.172650] ? hrtimer_interrupt+0x128/0x2c0 [ 9188.176057] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9188.179533] ? apic_timer_interrupt+0xf/0x20 [ 9188.182186] [ 9188.184872] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9188.186258] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9188.187696] ? dump_stack+0xf6/0x10e [ 9188.193312] do_raw_spin_lock+0x177/0x190 [ 9188.196031] _raw_spin_lock+0x21/0x30 [ 9188.201765] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9188.203361] obd_mod_free+0x46/0x140 [ptlrpc] [ 9188.206512] mdc_commit_open+0x91/0xb0 [mdc] [ 9188.209511] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9188.211963] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9188.215924] after_reply+0xe2c/0x1780 [ptlrpc] [ 9188.218480] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9188.221873] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9188.224374] ? do_wait_intr+0xf0/0xf0 [ 9188.226188] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9188.229989] kthread+0x1d1/0x200 [ 9188.231344] ? set_kthread_struct+0x70/0x70 [ 9188.233264] ret_from_fork+0x1f/0x30 [ 9192.415089] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7167s! [ 9192.420313] Showing busy workqueues and worker pools: [ 9192.422186] workqueue events: flags=0x0 [ 9192.423198] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9192.425325] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9192.427617] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9192.430995] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9192.435224] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9192.437564] pending: kfree_rcu_monitor [ 9192.439784] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9192.442749] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9192.450808] workqueue events_long: flags=0x0 [ 9192.452811] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9192.459260] pending: lu_objects_flush [obdclass] [ 9192.461374] workqueue events_unbound: flags=0x2 [ 9192.463486] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9192.466381] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9192.470038] workqueue events_power_efficient: flags=0x80 [ 9192.472501] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9192.475463] pending: neigh_periodic_work, check_lifetime [ 9192.477628] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9192.479559] pending: do_cache_clean, neigh_periodic_work [ 9192.481792] workqueue mm_percpu_wq: flags=0x8 [ 9192.483439] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9192.486210] pending: lru_add_drain_per_cpu [ 9192.489022] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9192.491355] pending: vmstat_update, lru_add_drain_per_cpu [ 9192.493371] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9192.495727] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9192.498247] workqueue writeback: flags=0x4a [ 9192.499474] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9192.501766] pending: wb_workfn, wb_workfn [ 9192.503476] workqueue xprtiod: flags=0xa [ 9192.505013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9192.507188] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9192.510630] workqueue ipv6_addrconf: flags=0x40008 [ 9192.513364] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9192.517057] pending: addrconf_verify_work [ 9192.519041] workqueue knbd0-recv: flags=0x1a [ 9192.520779] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9192.523865] in-flight: 655:recv_work [ 9192.525589] workqueue ptlrpc_pinger: flags=0x2 [ 9192.527470] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9192.530321] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9192.533398] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7167s workers=4 idle: 36396 38 11188 [ 9192.537775] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9183s workers=2 idle: 658 [ 9200.051965] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 9200.059750] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9200.085380] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9200.090366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9200.094384] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9200.097625] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9200.100742] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9200.108668] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9200.111747] RAX: 00000000000069fc RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9200.114985] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9200.118228] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9200.121209] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9200.124635] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9200.127836] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9200.131486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9200.134031] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9200.137124] Call Trace: [ 9200.138235] [ 9200.139201] ? show_regs.cold.9+0x22/0x2f [ 9200.141104] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9200.143379] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9200.145378] ? watchdog+0x40/0x40 [ 9200.146651] ? hrtimer_interrupt+0x128/0x2c0 [ 9200.148893] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9200.151259] ? apic_timer_interrupt+0xf/0x20 [ 9200.152908] [ 9200.153870] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9200.156350] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9200.158788] do_raw_spin_lock+0x177/0x190 [ 9200.160681] _raw_spin_lock+0x21/0x30 [ 9200.162225] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9200.164931] process_one_work+0x2c8/0x700 [ 9200.166780] worker_thread+0x296/0x6e0 [ 9200.168618] ? rescuer_thread+0x570/0x570 [ 9200.170345] kthread+0x1d1/0x200 [ 9200.171817] ? set_kthread_struct+0x70/0x70 [ 9200.173331] ret_from_fork+0x1f/0x30 [ 9208.052633] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 9208.060645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9208.094759] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9208.097950] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9208.101404] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9208.109056] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9208.118420] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9208.121437] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9208.124495] RDX: 0000000000006077 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9208.127671] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9208.130429] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9208.133063] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9208.136787] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9208.142813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9208.144924] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9208.150709] Call Trace: [ 9208.152690] [ 9208.154483] ? show_regs.cold.9+0x22/0x2f [ 9208.156227] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9208.161731] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9208.163531] ? watchdog+0x40/0x40 [ 9208.165822] ? hrtimer_interrupt+0x128/0x2c0 [ 9208.167085] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9208.169008] ? apic_timer_interrupt+0xf/0x20 [ 9208.170579] [ 9208.173738] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9208.175751] do_raw_spin_lock+0x177/0x190 [ 9208.177154] _raw_spin_lock+0x21/0x30 [ 9208.178495] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9208.180904] ? do_raw_spin_unlock+0x75/0x190 [ 9208.182940] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9208.185119] ? do_wait_intr+0xf0/0xf0 [ 9208.186688] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9208.188797] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9208.191001] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9208.193426] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9208.197897] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9208.199846] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9208.203930] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9208.207112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9208.209403] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9208.215113] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9208.217575] lmv_intent_open+0x338/0x1150 [lmv] [ 9208.221243] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9208.226211] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9208.228733] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9208.231014] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9208.233250] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9208.236156] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9208.238242] ? do_raw_spin_unlock+0x75/0x190 [ 9208.240212] ? _raw_spin_unlock+0x12/0x30 [ 9208.243232] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9208.245178] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9208.247055] ll_atomic_open+0x36a/0xd70 [lustre] [ 9208.249047] lookup_open+0x356/0xb40 [ 9208.250417] do_last+0x313/0xfc0 [ 9208.251611] ? nd_jump_root+0xe5/0x160 [ 9208.252941] ? path_init+0x437/0x520 [ 9208.254028] path_openat+0xf7/0x500 [ 9208.261170] ? do_raw_spin_unlock+0x75/0x190 [ 9208.262974] do_filp_open+0x99/0x140 [ 9208.265191] ? getname_flags+0x6e/0x330 [ 9208.268366] ? __check_object_size+0xff/0x256 [ 9208.270156] ? do_raw_spin_unlock+0x75/0x190 [ 9208.272082] ? _raw_spin_unlock+0x12/0x30 [ 9208.273411] do_sys_openat2+0x2b4/0x410 [ 9208.274425] do_sys_open+0x73/0xa0 [ 9208.275313] __x64_sys_openat+0x24/0x30 [ 9208.276350] do_syscall_64+0xc1/0x440 [ 9208.277643] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9208.279139] RIP: 0033:0x7fdc0afbf332 [ 9208.280353] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9208.286262] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9208.289174] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9208.291418] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9208.293938] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9208.296884] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9208.299361] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9212.061812] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 9212.072347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9212.117198] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9212.126144] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9212.131604] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9212.135362] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9212.144724] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9212.148755] RAX: 000000000000610f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9212.153340] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9212.156758] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9212.159860] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9212.162906] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9212.167187] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9212.176746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9212.183444] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9212.193807] Call Trace: [ 9212.195034] [ 9212.197578] ? show_regs.cold.9+0x22/0x2f [ 9212.202277] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9212.203891] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9212.223455] ? watchdog+0x40/0x40 [ 9212.228160] ? hrtimer_interrupt+0x128/0x2c0 [ 9212.235531] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9212.240821] ? apic_timer_interrupt+0xf/0x20 [ 9212.243598] [ 9212.245235] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9212.247063] do_raw_spin_lock+0x177/0x190 [ 9212.248514] _raw_spin_lock+0x21/0x30 [ 9212.255019] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9212.258591] ? do_raw_spin_unlock+0x75/0x190 [ 9212.261595] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9212.265155] ? do_wait_intr+0xf0/0xf0 [ 9212.268276] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9212.271855] mdc_close+0x53b/0xd70 [mdc] [ 9212.280112] lmv_close+0x18d/0x490 [lmv] [ 9212.284026] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9212.291147] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9212.299225] ll_md_close+0x274/0x800 [lustre] [ 9212.302491] ll_file_release+0x128/0xc50 [lustre] [ 9212.306949] __fput+0x10d/0x440 [ 9212.313078] ____fput+0x12/0x20 [ 9212.315337] task_work_run+0xc6/0x110 [ 9212.319520] exit_to_usermode_loop+0x1dd/0x1f0 [ 9212.330120] do_syscall_64+0x3d6/0x440 [ 9212.334969] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9212.345968] RIP: 0033:0x7fdc0afbfcd8 [ 9212.347149] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9212.360304] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9212.366120] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9212.371284] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9212.375018] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9212.377343] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9212.387637] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9220.048009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 9220.053765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9220.098903] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9220.118817] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9220.129700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9220.134579] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9220.161721] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9220.172545] RAX: 0000000000004ce8 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9220.180526] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9220.184790] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9220.192122] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9220.198353] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9220.203851] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9220.206786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9220.213889] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9220.218661] Call Trace: [ 9220.219782] [ 9220.220504] ? show_regs.cold.9+0x22/0x2f [ 9220.221855] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9220.226741] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9220.231429] ? watchdog+0x40/0x40 [ 9220.234020] ? hrtimer_interrupt+0x128/0x2c0 [ 9220.239494] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9220.242529] ? apic_timer_interrupt+0xf/0x20 [ 9220.247004] [ 9220.247922] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9220.254724] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9220.257424] ? dump_stack+0xf6/0x10e [ 9220.259899] do_raw_spin_lock+0x177/0x190 [ 9220.268674] _raw_spin_lock+0x21/0x30 [ 9220.272531] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9220.277259] obd_mod_free+0x46/0x140 [ptlrpc] [ 9220.280283] mdc_commit_open+0x91/0xb0 [mdc] [ 9220.286294] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9220.291649] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9220.299548] after_reply+0xe2c/0x1780 [ptlrpc] [ 9220.309393] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9220.316370] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9220.320203] ? do_wait_intr+0xf0/0xf0 [ 9220.323990] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9220.332192] kthread+0x1d1/0x200 [ 9220.338395] ? set_kthread_struct+0x70/0x70 [ 9220.339924] ret_from_fork+0x1f/0x30 [ 9222.625014] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7197s! [ 9222.637902] Showing busy workqueues and worker pools: [ 9222.650251] workqueue events: flags=0x0 [ 9222.651870] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9222.658978] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9222.668076] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9222.673504] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9222.683282] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9222.704167] pending: kfree_rcu_monitor [ 9222.707133] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9222.711892] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9222.718367] workqueue events_long: flags=0x0 [ 9222.720358] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9222.723984] pending: lu_objects_flush [obdclass] [ 9222.726549] workqueue events_unbound: flags=0x2 [ 9222.729232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9222.732075] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9222.736270] workqueue events_power_efficient: flags=0x80 [ 9222.738576] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9222.743837] pending: neigh_periodic_work, check_lifetime [ 9222.747448] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9222.754628] pending: do_cache_clean, neigh_periodic_work [ 9222.765460] workqueue mm_percpu_wq: flags=0x8 [ 9222.767460] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9222.775469] pending: lru_add_drain_per_cpu [ 9222.784302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9222.788042] pending: vmstat_update, lru_add_drain_per_cpu [ 9222.791739] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9222.798878] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9222.801680] workqueue writeback: flags=0x4a [ 9222.804615] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9222.811675] pending: wb_workfn, wb_workfn [ 9222.816666] workqueue xprtiod: flags=0xa [ 9222.819556] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9222.822129] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9222.831381] workqueue ipv6_addrconf: flags=0x40008 [ 9222.833685] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9222.838696] pending: addrconf_verify_work [ 9222.840325] workqueue knbd0-recv: flags=0x1a [ 9222.842777] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9222.845643] in-flight: 655:recv_work [ 9222.847160] workqueue ptlrpc_pinger: flags=0x2 [ 9222.849402] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9222.854950] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9222.857599] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7197s workers=4 idle: 36396 38 11188 [ 9222.860609] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9213s workers=2 idle: 658 [ 9228.051570] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u8:1:36] [ 9228.053292] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9228.072680] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9228.079284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9228.082015] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9228.087012] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.092532] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9228.105774] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9228.108158] RAX: 0000000000007829 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9228.111994] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9228.114085] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9228.116219] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9228.118216] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9228.123444] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9228.126445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9228.130463] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9228.133690] Call Trace: [ 9228.135839] [ 9228.136463] ? show_regs.cold.9+0x22/0x2f [ 9228.137685] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9228.142404] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9228.143650] ? watchdog+0x40/0x40 [ 9228.144664] ? hrtimer_interrupt+0x128/0x2c0 [ 9228.146261] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9228.148722] ? apic_timer_interrupt+0xf/0x20 [ 9228.149837] [ 9228.150427] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.152686] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9228.154191] do_raw_spin_lock+0x177/0x190 [ 9228.155326] _raw_spin_lock+0x21/0x30 [ 9228.156256] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9228.157946] process_one_work+0x2c8/0x700 [ 9228.159048] worker_thread+0x296/0x6e0 [ 9228.160092] ? rescuer_thread+0x570/0x570 [ 9228.161288] kthread+0x1d1/0x200 [ 9228.162128] ? set_kthread_struct+0x70/0x70 [ 9228.163341] ret_from_fork+0x1f/0x30 [ 9236.052530] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 9236.058409] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9236.085902] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9236.092645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9236.095913] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9236.097502] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9236.118356] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9236.123151] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9236.125750] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9236.131196] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9236.135880] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9236.138149] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9236.144234] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9236.147941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9236.150374] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9236.154278] Call Trace: [ 9236.155443] [ 9236.157254] ? show_regs.cold.9+0x22/0x2f [ 9236.159302] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9236.161538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9236.163653] ? watchdog+0x40/0x40 [ 9236.165252] ? hrtimer_interrupt+0x128/0x2c0 [ 9236.167301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9236.169378] ? apic_timer_interrupt+0xf/0x20 [ 9236.171528] [ 9236.172557] ? native_safe_halt+0x2e/0x50 [ 9236.175398] kvm_wait+0xcd/0xe0 [ 9236.177277] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9236.179402] do_raw_spin_lock+0x177/0x190 [ 9236.183835] _raw_spin_lock+0x21/0x30 [ 9236.186419] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9236.190437] ? do_raw_spin_unlock+0x75/0x190 [ 9236.192353] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9236.196900] ? do_wait_intr+0xf0/0xf0 [ 9236.198090] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9236.201022] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9236.207051] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9236.211372] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9236.216179] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9236.218432] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9236.220489] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9236.224596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9236.230232] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9236.237124] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9236.240337] lmv_intent_open+0x338/0x1150 [lmv] [ 9236.242091] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9236.244305] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9236.250410] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9236.252178] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9236.260664] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9236.262427] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9236.272963] ? do_raw_spin_unlock+0x75/0x190 [ 9236.278062] ? _raw_spin_unlock+0x12/0x30 [ 9236.280425] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9236.283085] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9236.285468] ll_atomic_open+0x36a/0xd70 [lustre] [ 9236.290586] lookup_open+0x356/0xb40 [ 9236.293921] do_last+0x313/0xfc0 [ 9236.296546] ? nd_jump_root+0xe5/0x160 [ 9236.301406] ? path_init+0x437/0x520 [ 9236.306631] path_openat+0xf7/0x500 [ 9236.309578] ? do_raw_spin_unlock+0x75/0x190 [ 9236.316626] do_filp_open+0x99/0x140 [ 9236.319473] ? getname_flags+0x6e/0x330 [ 9236.322121] ? __check_object_size+0xff/0x256 [ 9236.328070] ? do_raw_spin_unlock+0x75/0x190 [ 9236.329453] ? _raw_spin_unlock+0x12/0x30 [ 9236.330673] do_sys_openat2+0x2b4/0x410 [ 9236.334345] do_sys_open+0x73/0xa0 [ 9236.335331] __x64_sys_openat+0x24/0x30 [ 9236.336986] do_syscall_64+0xc1/0x440 [ 9236.338414] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9236.341450] RIP: 0033:0x7fdc0afbf332 [ 9236.343048] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9236.349960] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9236.352980] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9236.355153] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9236.357174] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9236.359975] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9236.362505] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9240.054029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 9240.056956] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9240.085154] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9240.089536] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9240.094236] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9240.097735] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9240.120277] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9240.127888] RAX: 000000000000487a RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9240.132027] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9240.136066] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9240.141009] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9240.144478] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9240.147107] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9240.149264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9240.151522] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9240.154744] Call Trace: [ 9240.155778] [ 9240.156568] ? show_regs.cold.9+0x22/0x2f [ 9240.158255] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9240.160168] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9240.167215] ? watchdog+0x40/0x40 [ 9240.167982] ? hrtimer_interrupt+0x128/0x2c0 [ 9240.172468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9240.174029] ? apic_timer_interrupt+0xf/0x20 [ 9240.175351] [ 9240.176011] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9240.178743] do_raw_spin_lock+0x177/0x190 [ 9240.179997] _raw_spin_lock+0x21/0x30 [ 9240.181029] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9240.182677] ? do_raw_spin_unlock+0x75/0x190 [ 9240.184063] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9240.186018] ? do_wait_intr+0xf0/0xf0 [ 9240.187164] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9240.188800] mdc_close+0x53b/0xd70 [mdc] [ 9240.189855] lmv_close+0x18d/0x490 [lmv] [ 9240.191189] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9240.193562] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9240.196755] ll_md_close+0x274/0x800 [lustre] [ 9240.198215] ll_file_release+0x128/0xc50 [lustre] [ 9240.200263] __fput+0x10d/0x440 [ 9240.202104] ____fput+0x12/0x20 [ 9240.203884] task_work_run+0xc6/0x110 [ 9240.205109] exit_to_usermode_loop+0x1dd/0x1f0 [ 9240.206484] do_syscall_64+0x3d6/0x440 [ 9240.207518] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9240.208705] RIP: 0033:0x7fdc0afbfcd8 [ 9240.209679] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9240.215796] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9240.218379] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9240.220700] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9240.224984] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9240.228420] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9240.231761] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9248.048057] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 9248.052074] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9248.071415] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9248.075041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9248.078202] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9248.079722] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9248.086213] RSP: 0018:ffffba8e41bf7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9248.088991] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9248.091416] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9248.093820] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9248.096171] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9248.098450] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9248.100755] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9248.103206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9248.104877] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9248.106966] Call Trace: [ 9248.107835] [ 9248.108832] ? show_regs.cold.9+0x22/0x2f [ 9248.110090] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9248.111510] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9248.115271] ? watchdog+0x40/0x40 [ 9248.116443] ? hrtimer_interrupt+0x128/0x2c0 [ 9248.117873] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9248.119438] ? apic_timer_interrupt+0xf/0x20 [ 9248.121264] [ 9248.122152] ? native_safe_halt+0x2e/0x50 [ 9248.123816] kvm_wait+0xcd/0xe0 [ 9248.125058] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9248.126708] ? dump_stack+0xf6/0x10e [ 9248.128093] do_raw_spin_lock+0x177/0x190 [ 9248.129463] _raw_spin_lock+0x21/0x30 [ 9248.130778] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9248.132645] obd_mod_free+0x46/0x140 [ptlrpc] [ 9248.134222] mdc_commit_open+0x91/0xb0 [mdc] [ 9248.136098] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9248.138050] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9248.140253] after_reply+0xe2c/0x1780 [ptlrpc] [ 9248.141975] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9248.143859] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9248.145397] ? do_wait_intr+0xf0/0xf0 [ 9248.146715] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9248.148636] kthread+0x1d1/0x200 [ 9248.149667] ? set_kthread_struct+0x70/0x70 [ 9248.150983] ret_from_fork+0x1f/0x30 [ 9253.343058] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7228s! [ 9253.347147] Showing busy workqueues and worker pools: [ 9253.352577] workqueue events: flags=0x0 [ 9253.355838] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9253.359990] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9253.363786] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9253.369635] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9253.378391] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9253.387349] pending: kfree_rcu_monitor [ 9253.390415] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9253.396417] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9253.409826] workqueue events_long: flags=0x0 [ 9253.412195] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9253.416245] pending: lu_objects_flush [obdclass] [ 9253.419231] workqueue events_unbound: flags=0x2 [ 9253.420965] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9253.424724] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9253.429785] workqueue events_power_efficient: flags=0x80 [ 9253.437752] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9253.439973] pending: neigh_periodic_work, check_lifetime [ 9253.443389] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9253.449205] pending: do_cache_clean, neigh_periodic_work [ 9253.451344] workqueue mm_percpu_wq: flags=0x8 [ 9253.452935] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9253.455569] pending: lru_add_drain_per_cpu [ 9253.456937] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9253.463384] pending: vmstat_update, lru_add_drain_per_cpu [ 9253.467746] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9253.473575] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9253.475877] workqueue writeback: flags=0x4a [ 9253.477268] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9253.481831] pending: wb_workfn, wb_workfn [ 9253.488390] workqueue xprtiod: flags=0xa [ 9253.494427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9253.502302] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9253.512782] workqueue ipv6_addrconf: flags=0x40008 [ 9253.516314] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9253.519818] pending: addrconf_verify_work [ 9253.522309] workqueue knbd0-recv: flags=0x1a [ 9253.526387] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9253.529330] in-flight: 655:recv_work [ 9253.537176] workqueue ptlrpc_pinger: flags=0x2 [ 9253.539910] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9253.544201] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9253.551475] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7228s workers=4 idle: 36396 38 11188 [ 9253.559142] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9244s workers=2 idle: 658 [ 9256.052012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 9256.057229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9256.074896] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9256.079912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9256.083845] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9256.087272] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.089950] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9256.098960] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9256.102638] RAX: 0000000000007f83 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9256.105353] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9256.108015] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9256.111053] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9256.114002] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9256.116904] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9256.119961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9256.122750] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9256.125390] Call Trace: [ 9256.126391] [ 9256.127057] ? show_regs.cold.9+0x22/0x2f [ 9256.128548] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9256.129934] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9256.131742] ? watchdog+0x40/0x40 [ 9256.132791] ? hrtimer_interrupt+0x128/0x2c0 [ 9256.134316] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9256.135957] ? apic_timer_interrupt+0xf/0x20 [ 9256.137636] [ 9256.138426] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.140597] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9256.142913] do_raw_spin_lock+0x177/0x190 [ 9256.144620] _raw_spin_lock+0x21/0x30 [ 9256.146711] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9256.149507] process_one_work+0x2c8/0x700 [ 9256.150837] worker_thread+0x296/0x6e0 [ 9256.152299] ? rescuer_thread+0x570/0x570 [ 9256.153724] kthread+0x1d1/0x200 [ 9256.154958] ? set_kthread_struct+0x70/0x70 [ 9256.156624] ret_from_fork+0x1f/0x30 [ 9264.052343] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 9264.055903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9264.082557] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9264.089778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9264.094239] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9264.097910] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9264.106328] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9264.110007] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9264.114387] RDX: 0000000000005e29 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9264.117160] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9264.121494] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9264.123997] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9264.126454] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9264.129152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9264.131584] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9264.134099] Call Trace: [ 9264.134989] [ 9264.135820] ? show_regs.cold.9+0x22/0x2f [ 9264.137447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9264.138990] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9264.140598] ? watchdog+0x40/0x40 [ 9264.142026] ? hrtimer_interrupt+0x128/0x2c0 [ 9264.143691] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9264.145460] ? apic_timer_interrupt+0xf/0x20 [ 9264.147190] [ 9264.148202] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9264.150971] do_raw_spin_lock+0x177/0x190 [ 9264.152496] _raw_spin_lock+0x21/0x30 [ 9264.153890] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9264.156626] ? do_raw_spin_unlock+0x75/0x190 [ 9264.158736] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9264.160618] ? do_wait_intr+0xf0/0xf0 [ 9264.162162] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9264.164171] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9264.166205] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9264.168253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9264.170503] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9264.172452] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9264.174077] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9264.175737] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9264.177833] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9264.179947] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9264.182428] lmv_intent_open+0x338/0x1150 [lmv] [ 9264.184026] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9264.186071] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9264.188077] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9264.189820] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9264.191680] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9264.193520] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9264.195434] ? do_raw_spin_unlock+0x75/0x190 [ 9264.196720] ? _raw_spin_unlock+0x12/0x30 [ 9264.198067] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9264.199725] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9264.202206] ll_atomic_open+0x36a/0xd70 [lustre] [ 9264.203871] lookup_open+0x356/0xb40 [ 9264.205075] do_last+0x313/0xfc0 [ 9264.206158] ? nd_jump_root+0xe5/0x160 [ 9264.207961] ? path_init+0x437/0x520 [ 9264.209256] path_openat+0xf7/0x500 [ 9264.210619] ? do_raw_spin_unlock+0x75/0x190 [ 9264.212076] do_filp_open+0x99/0x140 [ 9264.213256] ? getname_flags+0x6e/0x330 [ 9264.214883] ? __check_object_size+0xff/0x256 [ 9264.216933] ? do_raw_spin_unlock+0x75/0x190 [ 9264.218117] ? _raw_spin_unlock+0x12/0x30 [ 9264.219398] do_sys_openat2+0x2b4/0x410 [ 9264.220614] do_sys_open+0x73/0xa0 [ 9264.221859] __x64_sys_openat+0x24/0x30 [ 9264.223531] do_syscall_64+0xc1/0x440 [ 9264.225294] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9264.226931] RIP: 0033:0x7fdc0afbf332 [ 9264.228104] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9264.235550] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9264.238375] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9264.241119] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9264.243821] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9264.246687] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9264.249746] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9268.055694] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 9268.071026] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9268.121639] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9268.129493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9268.136493] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9268.138395] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9268.148521] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9268.151994] RAX: 0000000000005040 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9268.155030] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9268.157148] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9268.161092] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9268.163629] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9268.166816] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9268.169829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9268.171496] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9268.173668] Call Trace: [ 9268.174461] [ 9268.175051] ? show_regs.cold.9+0x22/0x2f [ 9268.176400] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9268.177713] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9268.179308] ? watchdog+0x40/0x40 [ 9268.180769] ? hrtimer_interrupt+0x128/0x2c0 [ 9268.182609] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9268.184963] ? apic_timer_interrupt+0xf/0x20 [ 9268.187104] [ 9268.188060] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9268.190649] do_raw_spin_lock+0x177/0x190 [ 9268.192655] _raw_spin_lock+0x21/0x30 [ 9268.197586] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9268.200852] ? do_raw_spin_unlock+0x75/0x190 [ 9268.203691] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9268.206894] ? do_wait_intr+0xf0/0xf0 [ 9268.209015] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9268.215692] mdc_close+0x53b/0xd70 [mdc] [ 9268.217204] lmv_close+0x18d/0x490 [lmv] [ 9268.219579] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9268.221782] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9268.223689] ll_md_close+0x274/0x800 [lustre] [ 9268.225488] ll_file_release+0x128/0xc50 [lustre] [ 9268.227430] __fput+0x10d/0x440 [ 9268.228666] ____fput+0x12/0x20 [ 9268.229841] task_work_run+0xc6/0x110 [ 9268.231262] exit_to_usermode_loop+0x1dd/0x1f0 [ 9268.236404] do_syscall_64+0x3d6/0x440 [ 9268.237925] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9268.239959] RIP: 0033:0x7fdc0afbfcd8 [ 9268.243061] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9268.251145] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9268.254718] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9268.257512] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9268.260111] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9268.262993] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9268.268746] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9279.513128] rcu: INFO: rcu_sched self-detected stall on CPU [ 9279.515511] rcu: 0-...!: (4552315 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 9279.520132] (t=7260170 jiffies g=270761 q=4460) [ 9279.523147] rcu: rcu_sched kthread starved for 7255102 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 9279.528343] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9279.532517] rcu: RCU grace-period kthread stack dump: [ 9279.535261] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9279.541103] Call Trace: [ 9279.542086] __schedule+0x351/0xcb0 [ 9279.543788] schedule+0xc0/0x180 [ 9279.545519] schedule_timeout+0xb4/0x190 [ 9279.547877] ? do_raw_spin_unlock+0x75/0x190 [ 9279.550824] ? __next_timer_interrupt+0x160/0x160 [ 9279.553146] ? prepare_to_swait_event+0xf3/0x140 [ 9279.555180] rcu_gp_kthread+0x8dd/0x15b0 [ 9279.556859] ? force_qs_rnp+0x2c0/0x2c0 [ 9279.558595] kthread+0x1d1/0x200 [ 9279.559936] ? set_kthread_struct+0x70/0x70 [ 9279.561753] ret_from_fork+0x1f/0x30 [ 9279.563338] rcu: Stack dump where RCU GP kthread last ran: [ 9279.565790] Sending NMI from CPU 0 to CPUs 3: [ 9279.570994] NMI backtrace for cpu 3 [ 9279.571001] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9279.571004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9279.571005] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9279.571006] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9279.571014] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 [ 9279.571018] RAX: 000000000000700b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9279.571019] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9279.571021] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9279.571022] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9279.571024] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9279.571026] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9279.571027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9279.571029] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9279.571030] Call Trace: [ 9279.571031] [ 9279.571032] ? show_regs.cold.9+0x22/0x2f [ 9279.571033] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9279.571034] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9279.571035] ? nmi_handle+0xbd/0x210 [ 9279.571037] ? default_do_nmi+0x4b/0x1b0 [ 9279.571038] ? do_nmi+0x280/0x380 [ 9279.571039] ? end_repeat_nmi+0x16/0x1f [ 9279.571040] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9279.571041] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9279.571043] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9279.571044] [ 9279.571045] do_raw_spin_lock+0x177/0x190 [ 9279.571045] _raw_spin_lock+0x21/0x30 [ 9279.571047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9279.571048] ? do_raw_spin_unlock+0x75/0x190 [ 9279.571049] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9279.571050] ? do_wait_intr+0xf0/0xf0 [ 9279.571051] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9279.571052] mdc_close+0x53b/0xd70 [mdc] [ 9279.571053] lmv_close+0x18d/0x490 [lmv] [ 9279.571055] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9279.571056] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9279.571057] ll_md_close+0x274/0x800 [lustre] [ 9279.571058] ll_file_release+0x128/0xc50 [lustre] [ 9279.571059] __fput+0x10d/0x440 [ 9279.571060] ____fput+0x12/0x20 [ 9279.571061] task_work_run+0xc6/0x110 [ 9279.571062] exit_to_usermode_loop+0x1dd/0x1f0 [ 9279.571066] do_syscall_64+0x3d6/0x440 [ 9279.571068] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9279.571069] RIP: 0033:0x7fdc0afbfcd8 [ 9279.571070] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9279.571073] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9279.571075] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9279.571077] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9279.571079] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9279.571081] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9279.571082] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9279.571569] NMI backtrace for cpu 0 [ 9279.725951] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9279.731425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9279.735549] Call Trace: [ 9279.736933] [ 9279.737726] ? dump_stack+0xbb/0x10e [ 9279.739456] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9279.741826] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9279.743641] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9279.745670] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9279.747696] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9279.749565] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9279.750768] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9279.752637] ? update_process_times+0x6d/0xc0 [ 9279.754561] ? tick_sched_handle.isra.18+0x27/0x90 [ 9279.756731] ? tick_sched_timer+0x53/0xb0 [ 9279.758773] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9279.760699] ? tick_sched_handle.isra.18+0x90/0x90 [ 9279.763006] ? hrtimer_interrupt+0x128/0x2c0 [ 9279.764925] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9279.766490] ? apic_timer_interrupt+0xf/0x20 [ 9279.768152] [ 9279.768892] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9279.771239] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9279.773482] ? dump_stack+0xf6/0x10e [ 9279.775305] ? do_raw_spin_lock+0x177/0x190 [ 9279.777190] ? _raw_spin_lock+0x21/0x30 [ 9279.779136] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9279.781777] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9279.783665] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9279.785510] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9279.788564] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9279.792167] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9279.794042] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9279.796888] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9279.799559] ? do_wait_intr+0xf0/0xf0 [ 9279.801598] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9279.804699] ? kthread+0x1d1/0x200 [ 9279.806079] ? set_kthread_struct+0x70/0x70 [ 9279.807955] ? ret_from_fork+0x1f/0x30 [ 9284.052418] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 9284.057736] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9284.068028] BUG: workqueue lockup - pool cpus=0-3 [ 9284.092804] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9284.092840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9284.095660] flags=0x4 nice=0 stuck for 7259s! [ 9284.095678] Showing busy workqueues and worker pools: [ 9284.095681] workqueue events: flags=0x0 [ 9284.095684] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9284.095699] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9284.096805] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9284.096817] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9284.097851] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9284.097862] pending: kfree_rcu_monitor [ 9284.097871] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9284.097879] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9284.099506] workqueue events_long: flags=0x0 [ 9284.099512] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9284.100049] pending: lu_objects_flush [obdclass] [ 9284.110441] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9284.110791] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9284.110803] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9284.110807] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9284.110811] RAX: 0000000000007e1b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9284.110814] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9284.110816] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9284.110818] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9284.110820] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9284.110823] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9284.110826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9284.110828] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9284.110838] Call Trace: [ 9284.110844] [ 9284.110849] ? show_regs.cold.9+0x22/0x2f [ 9284.110858] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9284.110864] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9284.110868] ? watchdog+0x40/0x40 [ 9284.110872] ? hrtimer_interrupt+0x128/0x2c0 [ 9284.110877] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9284.110885] ? apic_timer_interrupt+0xf/0x20 [ 9284.110890] [ 9284.110893] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9284.110899] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9284.110904] do_raw_spin_lock+0x177/0x190 [ 9284.110909] _raw_spin_lock+0x21/0x30 [ 9284.111811] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9284.112136] process_one_work+0x2c8/0x700 [ 9284.112144] worker_thread+0x296/0x6e0 [ 9284.112148] ? rescuer_thread+0x570/0x570 [ 9284.112152] kthread+0x1d1/0x200 [ 9284.112156] ? set_kthread_struct+0x70/0x70 [ 9284.112160] ret_from_fork+0x1f/0x30 [ 9284.301106] [ 9284.301713] workqueue events_unbound: flags=0x2 [ 9284.307014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9284.311178] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9284.316052] workqueue events_power_efficient: flags=0x80 [ 9284.318583] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9284.322369] pending: neigh_periodic_work, check_lifetime [ 9284.324933] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9284.328459] pending: do_cache_clean, neigh_periodic_work [ 9284.331934] workqueue mm_percpu_wq: flags=0x8 [ 9284.335158] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9284.337273] pending: lru_add_drain_per_cpu [ 9284.339480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9284.342688] pending: vmstat_update, lru_add_drain_per_cpu [ 9284.345621] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9284.348685] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9284.352015] workqueue writeback: flags=0x4a [ 9284.354167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9284.357749] pending: wb_workfn, wb_workfn [ 9284.359876] workqueue xprtiod: flags=0xa [ 9284.362244] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9284.365461] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9284.370276] workqueue ipv6_addrconf: flags=0x40008 [ 9284.372413] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9284.376089] pending: addrconf_verify_work [ 9284.378308] workqueue knbd0-recv: flags=0x1a [ 9284.380445] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9284.383272] in-flight: 655:recv_work [ 9284.385134] workqueue ptlrpc_pinger: flags=0x2 [ 9284.387619] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9284.396256] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9284.398581] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7259s workers=4 idle: 36396 38 11188 [ 9284.409749] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9274s workers=2 idle: 658 [ 9292.053834] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 9292.063734] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9292.100715] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9292.107171] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9292.113874] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9292.117103] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9292.127977] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9292.131269] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9292.133706] RDX: 0000000000005014 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9292.138872] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9292.142220] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9292.145996] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9292.149601] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9292.152890] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9292.155351] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9292.158292] Call Trace: [ 9292.159339] [ 9292.160123] ? show_regs.cold.9+0x22/0x2f [ 9292.162194] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9292.163891] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9292.166244] ? watchdog+0x40/0x40 [ 9292.172416] ? hrtimer_interrupt+0x128/0x2c0 [ 9292.174407] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9292.176199] ? apic_timer_interrupt+0xf/0x20 [ 9292.179291] [ 9292.181155] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9292.185910] do_raw_spin_lock+0x177/0x190 [ 9292.189597] _raw_spin_lock+0x21/0x30 [ 9292.191811] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9292.195320] ? do_raw_spin_unlock+0x75/0x190 [ 9292.197117] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9292.198657] ? do_wait_intr+0xf0/0xf0 [ 9292.199754] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9292.201985] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9292.205122] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9292.210045] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9292.212678] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9292.215352] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9292.217790] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9292.220088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9292.222645] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9292.224863] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9292.226772] lmv_intent_open+0x338/0x1150 [lmv] [ 9292.228698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9292.230592] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9292.232734] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9292.234408] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9292.236227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9292.238197] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9292.239869] ? do_raw_spin_unlock+0x75/0x190 [ 9292.242114] ? _raw_spin_unlock+0x12/0x30 [ 9292.243211] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9292.244844] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9292.246275] ll_atomic_open+0x36a/0xd70 [lustre] [ 9292.247705] lookup_open+0x356/0xb40 [ 9292.248659] do_last+0x313/0xfc0 [ 9292.249519] ? nd_jump_root+0xe5/0x160 [ 9292.250608] ? path_init+0x437/0x520 [ 9292.251497] path_openat+0xf7/0x500 [ 9292.252578] ? do_raw_spin_unlock+0x75/0x190 [ 9292.253712] do_filp_open+0x99/0x140 [ 9292.254697] ? getname_flags+0x6e/0x330 [ 9292.255679] ? __check_object_size+0xff/0x256 [ 9292.257128] ? do_raw_spin_unlock+0x75/0x190 [ 9292.258259] ? _raw_spin_unlock+0x12/0x30 [ 9292.259329] do_sys_openat2+0x2b4/0x410 [ 9292.260372] do_sys_open+0x73/0xa0 [ 9292.261648] __x64_sys_openat+0x24/0x30 [ 9292.262868] do_syscall_64+0xc1/0x440 [ 9292.263840] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9292.265206] RIP: 0033:0x7fdc0afbf332 [ 9292.266196] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9292.273616] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9292.278513] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9292.282505] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9292.286745] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9292.294122] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9292.303346] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9304.055797] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 9304.060347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9304.111087] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9304.122500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9304.126475] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9304.130817] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9304.140053] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9304.146847] RAX: 0000000000006d0c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9304.150408] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9304.156031] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9304.162141] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9304.168996] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9304.176955] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9304.182579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9304.184426] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9304.192942] Call Trace: [ 9304.195932] [ 9304.198284] ? show_regs.cold.9+0x22/0x2f [ 9304.202005] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9304.207587] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9304.209794] ? watchdog+0x40/0x40 [ 9304.211503] ? hrtimer_interrupt+0x128/0x2c0 [ 9304.212817] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9304.214388] ? apic_timer_interrupt+0xf/0x20 [ 9304.215748] [ 9304.216419] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9304.218394] do_raw_spin_lock+0x177/0x190 [ 9304.219902] _raw_spin_lock+0x21/0x30 [ 9304.221054] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9304.222945] ? do_raw_spin_unlock+0x75/0x190 [ 9304.224332] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9304.226432] ? do_wait_intr+0xf0/0xf0 [ 9304.231948] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9304.235478] mdc_close+0x53b/0xd70 [mdc] [ 9304.237302] lmv_close+0x18d/0x490 [lmv] [ 9304.239527] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9304.244648] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9304.250102] ll_md_close+0x274/0x800 [lustre] [ 9304.254685] ll_file_release+0x128/0xc50 [lustre] [ 9304.259241] __fput+0x10d/0x440 [ 9304.260348] ____fput+0x12/0x20 [ 9304.261430] task_work_run+0xc6/0x110 [ 9304.268355] exit_to_usermode_loop+0x1dd/0x1f0 [ 9304.274721] do_syscall_64+0x3d6/0x440 [ 9304.280317] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9304.283205] RIP: 0033:0x7fdc0afbfcd8 [ 9304.285162] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9304.293318] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9304.297312] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9304.304054] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9304.306876] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9304.308930] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9304.317085] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9312.053203] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 9312.056799] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 9312.060060] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9312.060304] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9312.060311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9312.060319] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9312.060765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.060777] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9312.060782] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9312.060786] RAX: 0000000000006117 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9312.060789] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9312.060791] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9312.060794] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9312.060796] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9312.060800] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9312.060803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9312.060881] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9312.060889] Call Trace: [ 9312.060895] [ 9312.060900] ? show_regs.cold.9+0x22/0x2f [ 9312.060909] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9312.060915] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9312.060920] ? watchdog+0x40/0x40 [ 9312.060925] ? hrtimer_interrupt+0x128/0x2c0 [ 9312.060929] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9312.060937] ? apic_timer_interrupt+0xf/0x20 [ 9312.060942] [ 9312.060945] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.060950] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9312.060955] do_raw_spin_lock+0x177/0x190 [ 9312.060960] _raw_spin_lock+0x21/0x30 [ 9312.060965] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9312.061981] process_one_work+0x2c8/0x700 [ 9312.061991] worker_thread+0x296/0x6e0 [ 9312.061995] ? rescuer_thread+0x570/0x570 [ 9312.061999] kthread+0x1d1/0x200 [ 9312.062003] ? set_kthread_struct+0x70/0x70 [ 9312.062007] ret_from_fork+0x1f/0x30 [ 9312.257973] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9312.287698] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9312.295266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9312.301674] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.305372] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9312.317843] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9312.324018] RAX: 0000000000007212 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9312.328700] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9312.331941] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9312.336381] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9312.342619] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9312.345197] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9312.349551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9312.353820] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9312.358810] Call Trace: [ 9312.360674] [ 9312.362249] ? show_regs.cold.9+0x22/0x2f [ 9312.363289] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9312.367005] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9312.368536] ? watchdog+0x40/0x40 [ 9312.370582] ? hrtimer_interrupt+0x128/0x2c0 [ 9312.374264] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9312.376328] ? apic_timer_interrupt+0xf/0x20 [ 9312.380604] [ 9312.381327] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.383854] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9312.387818] ? dump_stack+0xf6/0x10e [ 9312.389429] do_raw_spin_lock+0x177/0x190 [ 9312.390571] _raw_spin_lock+0x21/0x30 [ 9312.391670] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9312.396291] obd_mod_free+0x46/0x140 [ptlrpc] [ 9312.398470] mdc_commit_open+0x91/0xb0 [mdc] [ 9312.401760] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9312.405685] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9312.410323] after_reply+0xe2c/0x1780 [ptlrpc] [ 9312.414296] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9312.417942] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9312.423193] ? do_wait_intr+0xf0/0xf0 [ 9312.424465] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9312.426689] kthread+0x1d1/0x200 [ 9312.428354] ? set_kthread_struct+0x70/0x70 [ 9312.432708] ret_from_fork+0x1f/0x30 [ 9314.783365] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7289s! [ 9314.787270] Showing busy workqueues and worker pools: [ 9314.790141] workqueue events: flags=0x0 [ 9314.792094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9314.801037] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9314.806013] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9314.810933] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9314.821491] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9314.825133] pending: kfree_rcu_monitor [ 9314.827592] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9314.833682] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9314.843228] workqueue events_long: flags=0x0 [ 9314.846247] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9314.853465] pending: lu_objects_flush [obdclass] [ 9314.856213] workqueue events_unbound: flags=0x2 [ 9314.858728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9314.861934] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9314.866801] workqueue events_power_efficient: flags=0x80 [ 9314.872261] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9314.874374] pending: neigh_periodic_work, check_lifetime [ 9314.877931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9314.887240] pending: do_cache_clean, neigh_periodic_work [ 9314.891128] workqueue mm_percpu_wq: flags=0x8 [ 9314.893729] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9314.905596] pending: lru_add_drain_per_cpu [ 9314.909698] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9314.914046] pending: vmstat_update, lru_add_drain_per_cpu [ 9314.919751] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9314.924454] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9314.927511] workqueue writeback: flags=0x4a [ 9314.931307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9314.934282] pending: wb_workfn, wb_workfn [ 9314.936292] workqueue xprtiod: flags=0xa [ 9314.937684] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9314.939566] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9314.943338] workqueue ipv6_addrconf: flags=0x40008 [ 9314.948276] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9314.952796] pending: addrconf_verify_work [ 9314.954305] workqueue knbd0-recv: flags=0x1a [ 9314.956658] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9314.970233] in-flight: 655:recv_work [ 9314.973486] workqueue ptlrpc_pinger: flags=0x2 [ 9314.977498] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9314.980998] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9314.985847] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7290s workers=4 idle: 36396 38 11188 [ 9314.991013] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9305s workers=2 idle: 658 [ 9320.052048] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 9320.060847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9320.092414] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9320.096471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9320.101111] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9320.106278] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9320.127292] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9320.130882] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9320.138186] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9320.141794] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9320.147175] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9320.151219] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9320.155271] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9320.159786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9320.163248] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9320.166038] Call Trace: [ 9320.167470] [ 9320.168489] ? show_regs.cold.9+0x22/0x2f [ 9320.170388] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9320.172863] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9320.174369] ? watchdog+0x40/0x40 [ 9320.176902] ? hrtimer_interrupt+0x128/0x2c0 [ 9320.178878] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9320.185283] ? apic_timer_interrupt+0xf/0x20 [ 9320.187811] [ 9320.188575] ? native_safe_halt+0x2e/0x50 [ 9320.189863] kvm_wait+0xcd/0xe0 [ 9320.191429] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9320.194415] do_raw_spin_lock+0x177/0x190 [ 9320.197247] _raw_spin_lock+0x21/0x30 [ 9320.199610] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9320.202536] ? do_raw_spin_unlock+0x75/0x190 [ 9320.205065] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9320.208419] ? do_wait_intr+0xf0/0xf0 [ 9320.210297] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9320.213558] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9320.215846] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9320.219136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9320.221916] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9320.224710] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9320.227682] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9320.230311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9320.233114] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9320.236132] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9320.239602] lmv_intent_open+0x338/0x1150 [lmv] [ 9320.242233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9320.244571] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9320.246751] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9320.251034] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9320.253758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9320.257104] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9320.262940] ? do_raw_spin_unlock+0x75/0x190 [ 9320.265256] ? _raw_spin_unlock+0x12/0x30 [ 9320.269987] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9320.272589] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9320.275213] ll_atomic_open+0x36a/0xd70 [lustre] [ 9320.281480] lookup_open+0x356/0xb40 [ 9320.283861] do_last+0x313/0xfc0 [ 9320.285985] ? nd_jump_root+0xe5/0x160 [ 9320.287957] ? path_init+0x437/0x520 [ 9320.296503] path_openat+0xf7/0x500 [ 9320.298502] ? do_raw_spin_unlock+0x75/0x190 [ 9320.300770] do_filp_open+0x99/0x140 [ 9320.303127] ? getname_flags+0x6e/0x330 [ 9320.305207] ? __check_object_size+0xff/0x256 [ 9320.308096] ? do_raw_spin_unlock+0x75/0x190 [ 9320.310168] ? _raw_spin_unlock+0x12/0x30 [ 9320.312615] do_sys_openat2+0x2b4/0x410 [ 9320.314604] do_sys_open+0x73/0xa0 [ 9320.318643] __x64_sys_openat+0x24/0x30 [ 9320.320358] do_syscall_64+0xc1/0x440 [ 9320.322217] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9320.324379] RIP: 0033:0x7fdc0afbf332 [ 9320.326338] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9320.337713] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9320.341600] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9320.344514] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9320.347028] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9320.349724] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9320.352747] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9332.054011] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 9332.056322] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9332.095566] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9332.104234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9332.109227] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9332.113387] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9332.120910] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9332.124441] RAX: 000000000000658e RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9332.127848] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9332.131282] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9332.134881] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9332.139556] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9332.142987] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9332.146504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9332.149066] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9332.152166] Call Trace: [ 9332.153380] [ 9332.154372] ? show_regs.cold.9+0x22/0x2f [ 9332.155862] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9332.158179] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9332.159952] ? watchdog+0x40/0x40 [ 9332.161387] ? hrtimer_interrupt+0x128/0x2c0 [ 9332.163403] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9332.165790] ? apic_timer_interrupt+0xf/0x20 [ 9332.167843] [ 9332.168856] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9332.171330] do_raw_spin_lock+0x177/0x190 [ 9332.173215] _raw_spin_lock+0x21/0x30 [ 9332.174692] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9332.177532] ? do_raw_spin_unlock+0x75/0x190 [ 9332.179643] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9332.182255] ? do_wait_intr+0xf0/0xf0 [ 9332.184330] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9332.188233] mdc_close+0x53b/0xd70 [mdc] [ 9332.190229] lmv_close+0x18d/0x490 [lmv] [ 9332.191900] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9332.194575] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9332.196968] ll_md_close+0x274/0x800 [lustre] [ 9332.198951] ll_file_release+0x128/0xc50 [lustre] [ 9332.200684] __fput+0x10d/0x440 [ 9332.201694] ____fput+0x12/0x20 [ 9332.203289] task_work_run+0xc6/0x110 [ 9332.205205] exit_to_usermode_loop+0x1dd/0x1f0 [ 9332.207617] do_syscall_64+0x3d6/0x440 [ 9332.209308] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9332.211545] RIP: 0033:0x7fdc0afbfcd8 [ 9332.213188] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9332.221517] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9332.224997] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9332.228436] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9332.231717] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9332.234652] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9332.237038] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9340.051424] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 9340.054148] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 9340.056147] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9340.056228] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9340.056236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9340.056243] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9340.057329] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.057420] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9340.057424] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9340.057429] RAX: 000000000000137b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9340.057431] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9340.057433] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9340.057436] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9340.057438] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9340.057440] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9340.057443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9340.057445] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9340.057455] Call Trace: [ 9340.057460] [ 9340.057466] ? show_regs.cold.9+0x22/0x2f [ 9340.057475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9340.057482] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9340.057487] ? watchdog+0x40/0x40 [ 9340.057491] ? hrtimer_interrupt+0x128/0x2c0 [ 9340.057496] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9340.057504] ? apic_timer_interrupt+0xf/0x20 [ 9340.057509] [ 9340.057513] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.057590] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9340.057597] do_raw_spin_lock+0x177/0x190 [ 9340.057602] _raw_spin_lock+0x21/0x30 [ 9340.057607] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9340.070007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9340.070086] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9340.070100] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9340.070103] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.070116] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9340.070120] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9340.070125] RAX: 00000000000006be RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9340.070127] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9340.070129] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9340.070132] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9340.070138] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9340.070143] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9340.070146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9340.070148] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9340.070155] Call Trace: [ 9340.070160] [ 9340.070169] ? show_regs.cold.9+0x22/0x2f [ 9340.070178] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9340.070186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9340.070190] ? watchdog+0x40/0x40 [ 9340.070195] ? hrtimer_interrupt+0x128/0x2c0 [ 9340.070199] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9340.070207] ? apic_timer_interrupt+0xf/0x20 [ 9340.070211] [ 9340.070215] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.070220] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9340.070224] ? dump_stack+0xf6/0x10e [ 9340.070230] do_raw_spin_lock+0x177/0x190 [ 9340.070236] _raw_spin_lock+0x21/0x30 [ 9340.070242] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9340.070524] obd_mod_free+0x46/0x140 [ptlrpc] [ 9340.071457] mdc_commit_open+0x91/0xb0 [mdc] [ 9340.071495] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9340.071818] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9340.072195] after_reply+0xe2c/0x1780 [ptlrpc] [ 9340.072470] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9340.072794] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9340.073151] ? do_wait_intr+0xf0/0xf0 [ 9340.073168] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9340.073458] kthread+0x1d1/0x200 [ 9340.073465] ? set_kthread_struct+0x70/0x70 [ 9340.073469] ret_from_fork+0x1f/0x30 [ 9340.386417] process_one_work+0x2c8/0x700 [ 9340.388731] worker_thread+0x296/0x6e0 [ 9340.391246] ? rescuer_thread+0x570/0x570 [ 9340.393194] kthread+0x1d1/0x200 [ 9340.394891] ? set_kthread_struct+0x70/0x70 [ 9340.397162] ret_from_fork+0x1f/0x30 [ 9345.503205] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7320s! [ 9345.519151] Showing busy workqueues and worker pools: [ 9345.523427] workqueue events: flags=0x0 [ 9345.537585] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9345.539945] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9345.549986] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9345.560954] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9345.564617] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9345.567130] pending: kfree_rcu_monitor [ 9345.568495] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9345.570646] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9345.575620] workqueue events_long: flags=0x0 [ 9345.577206] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9345.582091] pending: lu_objects_flush [obdclass] [ 9345.583923] workqueue events_unbound: flags=0x2 [ 9345.585563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9345.587664] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9345.591626] workqueue events_power_efficient: flags=0x80 [ 9345.593426] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9345.596914] pending: neigh_periodic_work, check_lifetime [ 9345.598909] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9345.608642] pending: do_cache_clean, neigh_periodic_work [ 9345.611545] workqueue mm_percpu_wq: flags=0x8 [ 9345.613315] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9345.621635] pending: lru_add_drain_per_cpu [ 9345.623009] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9345.630137] pending: vmstat_update, lru_add_drain_per_cpu [ 9345.633534] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9345.638617] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9345.640631] workqueue writeback: flags=0x4a [ 9345.645398] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9345.650024] pending: wb_workfn, wb_workfn [ 9345.652554] workqueue xprtiod: flags=0xa [ 9345.654079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9345.662056] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9345.668162] workqueue ipv6_addrconf: flags=0x40008 [ 9345.671146] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9345.680773] pending: addrconf_verify_work [ 9345.683026] workqueue knbd0-recv: flags=0x1a [ 9345.685154] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9345.688354] in-flight: 655:recv_work [ 9345.691984] workqueue ptlrpc_pinger: flags=0x2 [ 9345.694702] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9345.701237] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9345.706210] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7320s workers=4 idle: 36396 38 11188 [ 9345.712609] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9336s workers=2 idle: 658 [ 9348.052020] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 9348.054229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9348.073092] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9348.077427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9348.080818] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9348.083123] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9348.090831] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9348.093687] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9348.096667] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9348.100233] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9348.103575] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9348.106422] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9348.109226] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9348.112365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9348.114512] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9348.117383] Call Trace: [ 9348.118373] [ 9348.119146] ? show_regs.cold.9+0x22/0x2f [ 9348.120674] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9348.122693] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9348.124553] ? watchdog+0x40/0x40 [ 9348.126162] ? hrtimer_interrupt+0x128/0x2c0 [ 9348.127831] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9348.130275] ? apic_timer_interrupt+0xf/0x20 [ 9348.132237] [ 9348.133278] ? native_safe_halt+0x2e/0x50 [ 9348.134986] kvm_wait+0xcd/0xe0 [ 9348.136439] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9348.138478] do_raw_spin_lock+0x177/0x190 [ 9348.140241] _raw_spin_lock+0x21/0x30 [ 9348.141595] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9348.143781] ? do_raw_spin_unlock+0x75/0x190 [ 9348.145640] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9348.147992] ? do_wait_intr+0xf0/0xf0 [ 9348.149558] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9348.151758] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9348.154502] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9348.156765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9348.159236] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9348.161133] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9348.163568] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9348.165644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9348.167960] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9348.170713] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9348.173556] lmv_intent_open+0x338/0x1150 [lmv] [ 9348.175555] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9348.178220] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9348.180824] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9348.182971] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9348.185413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9348.187896] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9348.190702] ? do_raw_spin_unlock+0x75/0x190 [ 9348.192380] ? _raw_spin_unlock+0x12/0x30 [ 9348.193843] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9348.195733] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9348.197732] ll_atomic_open+0x36a/0xd70 [lustre] [ 9348.199574] lookup_open+0x356/0xb40 [ 9348.200707] do_last+0x313/0xfc0 [ 9348.203298] ? nd_jump_root+0xe5/0x160 [ 9348.208758] ? path_init+0x437/0x520 [ 9348.211560] path_openat+0xf7/0x500 [ 9348.215904] ? do_raw_spin_unlock+0x75/0x190 [ 9348.219322] do_filp_open+0x99/0x140 [ 9348.220991] ? getname_flags+0x6e/0x330 [ 9348.223391] ? __check_object_size+0xff/0x256 [ 9348.225318] ? do_raw_spin_unlock+0x75/0x190 [ 9348.227417] ? _raw_spin_unlock+0x12/0x30 [ 9348.231820] do_sys_openat2+0x2b4/0x410 [ 9348.233609] do_sys_open+0x73/0xa0 [ 9348.237531] __x64_sys_openat+0x24/0x30 [ 9348.238797] do_syscall_64+0xc1/0x440 [ 9348.240474] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9348.242789] RIP: 0033:0x7fdc0afbf332 [ 9348.244401] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9348.251971] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9348.255161] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9348.258225] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9348.261292] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9348.264073] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9348.266510] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9360.057056] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 9360.063761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9360.113839] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9360.123031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9360.127637] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9360.132807] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9360.147544] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9360.153194] RAX: 0000000000005f0b RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9360.157806] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9360.163187] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9360.172730] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9360.176690] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9360.183326] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9360.188810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9360.198499] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9360.204782] Call Trace: [ 9360.205733] [ 9360.209617] ? show_regs.cold.9+0x22/0x2f [ 9360.216390] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9360.218234] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9360.225180] ? watchdog+0x40/0x40 [ 9360.233016] ? hrtimer_interrupt+0x128/0x2c0 [ 9360.235174] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9360.243457] ? apic_timer_interrupt+0xf/0x20 [ 9360.246057] [ 9360.247211] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9360.249851] do_raw_spin_lock+0x177/0x190 [ 9360.253009] _raw_spin_lock+0x21/0x30 [ 9360.254247] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9360.257360] ? do_raw_spin_unlock+0x75/0x190 [ 9360.259498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9360.262964] ? do_wait_intr+0xf0/0xf0 [ 9360.265951] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9360.269126] mdc_close+0x53b/0xd70 [mdc] [ 9360.272967] lmv_close+0x18d/0x490 [lmv] [ 9360.276218] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9360.281139] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9360.285124] ll_md_close+0x274/0x800 [lustre] [ 9360.292165] ll_file_release+0x128/0xc50 [lustre] [ 9360.298040] __fput+0x10d/0x440 [ 9360.299185] ____fput+0x12/0x20 [ 9360.301351] task_work_run+0xc6/0x110 [ 9360.305556] exit_to_usermode_loop+0x1dd/0x1f0 [ 9360.308235] do_syscall_64+0x3d6/0x440 [ 9360.311344] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9360.315372] RIP: 0033:0x7fdc0afbfcd8 [ 9360.318191] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9360.327238] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9360.334374] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9360.341127] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9360.344995] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9360.348527] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9360.351877] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9368.051147] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 9368.055200] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9368.074551] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9368.079188] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9368.082611] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9368.086660] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9368.088894] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9368.101345] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9368.108319] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9368.112120] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9368.117489] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9368.123635] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9368.127309] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9368.130779] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9368.134974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9368.137426] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9368.140726] Call Trace: [ 9368.141834] [ 9368.142832] ? show_regs.cold.9+0x22/0x2f [ 9368.144828] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9368.147283] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9368.150148] ? watchdog+0x40/0x40 [ 9368.152319] ? hrtimer_interrupt+0x128/0x2c0 [ 9368.154332] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9368.156695] ? apic_timer_interrupt+0xf/0x20 [ 9368.158754] [ 9368.159797] ? native_safe_halt+0x2e/0x50 [ 9368.161811] kvm_wait+0xcd/0xe0 [ 9368.163272] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9368.166934] do_raw_spin_lock+0x177/0x190 [ 9368.170377] _raw_spin_lock+0x21/0x30 [ 9368.175779] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9368.183149] process_one_work+0x2c8/0x700 [ 9368.186776] worker_thread+0x296/0x6e0 [ 9368.189287] ? rescuer_thread+0x570/0x570 [ 9368.190966] kthread+0x1d1/0x200 [ 9368.192749] ? set_kthread_struct+0x70/0x70 [ 9368.194790] ret_from_fork+0x1f/0x30 [ 9372.048144] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2388] [ 9372.051353] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9372.070534] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9372.074927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9372.078940] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9372.081068] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9372.088068] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9372.090797] RAX: 0000000000004597 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9372.093269] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9372.095774] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9372.098809] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9372.101530] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9372.103746] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9372.106956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9372.109086] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9372.111863] Call Trace: [ 9372.112893] [ 9372.113691] ? show_regs.cold.9+0x22/0x2f [ 9372.115252] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9372.117034] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9372.119297] ? watchdog+0x40/0x40 [ 9372.120639] ? hrtimer_interrupt+0x128/0x2c0 [ 9372.122243] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9372.123844] ? apic_timer_interrupt+0xf/0x20 [ 9372.125322] [ 9372.125934] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9372.127807] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9372.130007] ? dump_stack+0xf6/0x10e [ 9372.131494] do_raw_spin_lock+0x177/0x190 [ 9372.133105] _raw_spin_lock+0x21/0x30 [ 9372.134722] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9372.136970] obd_mod_free+0x46/0x140 [ptlrpc] [ 9372.138963] mdc_commit_open+0x91/0xb0 [mdc] [ 9372.140533] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9372.142273] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9372.144532] after_reply+0xe2c/0x1780 [ptlrpc] [ 9372.146318] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9372.148317] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9372.150046] ? do_wait_intr+0xf0/0xf0 [ 9372.151318] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9372.153523] kthread+0x1d1/0x200 [ 9372.154717] ? set_kthread_struct+0x70/0x70 [ 9372.156125] ret_from_fork+0x1f/0x30 [ 9376.052391] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 9376.060357] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9376.100809] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9376.111011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9376.117824] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9376.124673] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9376.155883] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9376.168761] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9376.174998] RDX: 00000000000050b4 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9376.181068] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9376.184522] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9376.188089] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9376.191705] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9376.196180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9376.200202] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9376.207778] Call Trace: [ 9376.209201] [ 9376.210281] ? show_regs.cold.9+0x22/0x2f [ 9376.212068] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9376.213963] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9376.215934] ? watchdog+0x40/0x40 [ 9376.217478] ? hrtimer_interrupt+0x128/0x2c0 [ 9376.219255] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9376.221577] ? apic_timer_interrupt+0xf/0x20 [ 9376.223644] [ 9376.224555] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9376.224649] BUG: workqueue lockup - pool [ 9376.226965] do_raw_spin_lock+0x177/0x190 [ 9376.227048] _raw_spin_lock+0x21/0x30 [ 9376.227061] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9376.227383] ? do_raw_spin_unlock+0x75/0x190 [ 9376.227389] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9376.227661] ? do_wait_intr+0xf0/0xf0 [ 9376.227671] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9376.227897] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9376.228192] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9376.228426] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9376.228533] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9376.228568] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9376.228941] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9376.228971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9376.229152] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9376.231938] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9376.231976] lmv_intent_open+0x338/0x1150 [lmv] [ 9376.232616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9376.236144] cpus=0-3 [ 9376.239431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9376.239557] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9376.239653] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9376.239749] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9376.239833] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9376.239923] ? do_raw_spin_unlock+0x75/0x190 [ 9376.239930] ? _raw_spin_unlock+0x12/0x30 [ 9376.239938] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9376.239966] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9376.240053] ll_atomic_open+0x36a/0xd70 [lustre] [ 9376.240148] lookup_open+0x356/0xb40 [ 9376.240157] do_last+0x313/0xfc0 [ 9376.240162] ? nd_jump_root+0xe5/0x160 [ 9376.240167] ? path_init+0x437/0x520 [ 9376.240172] path_openat+0xf7/0x500 [ 9376.240176] ? do_raw_spin_unlock+0x75/0x190 [ 9376.240226] do_filp_open+0x99/0x140 [ 9376.240235] ? getname_flags+0x6e/0x330 [ 9376.240241] ? __check_object_size+0xff/0x256 [ 9376.240249] ? do_raw_spin_unlock+0x75/0x190 [ 9376.240253] ? _raw_spin_unlock+0x12/0x30 [ 9376.240257] do_sys_openat2+0x2b4/0x410 [ 9376.240261] do_sys_open+0x73/0xa0 [ 9376.240265] __x64_sys_openat+0x24/0x30 [ 9376.240268] do_syscall_64+0xc1/0x440 [ 9376.240276] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9376.240283] RIP: 0033:0x7fdc0afbf332 [ 9376.240465] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9376.240472] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9376.240478] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9376.240480] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9376.240483] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9376.240485] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9376.240487] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9376.420218] flags=0x4 nice=0 stuck for 7351s! [ 9376.422132] Showing busy workqueues and worker pools: [ 9376.425083] workqueue events: flags=0x0 [ 9376.427132] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9376.430284] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9376.433269] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9376.436081] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9376.440125] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9376.442872] pending: kfree_rcu_monitor [ 9376.444794] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9376.448432] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9376.455699] workqueue events_long: flags=0x0 [ 9376.457963] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9376.461873] pending: lu_objects_flush [obdclass] [ 9376.464570] workqueue events_unbound: flags=0x2 [ 9376.469761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9376.474608] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9376.481808] workqueue events_power_efficient: flags=0x80 [ 9376.484327] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9376.488332] pending: neigh_periodic_work, check_lifetime [ 9376.491694] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9376.495566] pending: do_cache_clean, neigh_periodic_work [ 9376.499238] workqueue mm_percpu_wq: flags=0x8 [ 9376.501656] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9376.507679] pending: lru_add_drain_per_cpu [ 9376.510223] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9376.515778] pending: vmstat_update, lru_add_drain_per_cpu [ 9376.520359] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9376.524445] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9376.528033] workqueue writeback: flags=0x4a [ 9376.530622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9376.533393] pending: wb_workfn, wb_workfn [ 9376.535331] workqueue xprtiod: flags=0xa [ 9376.536953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9376.545658] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9376.552730] workqueue ipv6_addrconf: flags=0x40008 [ 9376.558602] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9376.565130] pending: addrconf_verify_work [ 9376.568291] workqueue knbd0-recv: flags=0x1a [ 9376.574063] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9376.582282] in-flight: 655:recv_work [ 9376.588322] workqueue ptlrpc_pinger: flags=0x2 [ 9376.590555] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9376.596259] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9376.603754] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7351s workers=4 idle: 36396 38 11188 [ 9376.614042] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9367s workers=2 idle: 658 [ 9388.054551] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:37520] [ 9388.062648] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9388.088578] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9388.094594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9388.098653] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9388.102330] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9388.114521] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9388.118795] RAX: 0000000000004e86 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9388.122079] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9388.125663] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9388.129071] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9388.132992] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9388.136672] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9388.143790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9388.146562] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9388.149755] Call Trace: [ 9388.151466] [ 9388.152485] ? show_regs.cold.9+0x22/0x2f [ 9388.154519] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9388.157019] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9388.159052] ? watchdog+0x40/0x40 [ 9388.160680] ? hrtimer_interrupt+0x128/0x2c0 [ 9388.162871] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9388.165383] ? apic_timer_interrupt+0xf/0x20 [ 9388.167416] [ 9388.168976] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9388.171789] do_raw_spin_lock+0x177/0x190 [ 9388.173797] _raw_spin_lock+0x21/0x30 [ 9388.175552] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9388.178114] ? do_raw_spin_unlock+0x75/0x190 [ 9388.179818] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9388.182332] ? do_wait_intr+0xf0/0xf0 [ 9388.183724] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9388.186328] mdc_close+0x53b/0xd70 [mdc] [ 9388.188132] lmv_close+0x18d/0x490 [lmv] [ 9388.190166] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9388.193028] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9388.196431] ll_md_close+0x274/0x800 [lustre] [ 9388.198793] ll_file_release+0x128/0xc50 [lustre] [ 9388.201435] __fput+0x10d/0x440 [ 9388.204673] ____fput+0x12/0x20 [ 9388.206494] task_work_run+0xc6/0x110 [ 9388.208379] exit_to_usermode_loop+0x1dd/0x1f0 [ 9388.210462] do_syscall_64+0x3d6/0x440 [ 9388.212028] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9388.214510] RIP: 0033:0x7fdc0afbfcd8 [ 9388.216514] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9388.229890] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9388.234485] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9388.239081] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9388.244400] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9388.251637] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9388.256061] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9396.052950] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 9396.055098] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9396.095030] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9396.102237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9396.107675] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9396.112256] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9396.115683] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9396.125226] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9396.129373] RAX: 0000000000007b0d RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9396.132897] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9396.136225] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9396.139029] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9396.148143] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9396.155683] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9396.162687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9396.168233] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9396.174195] Call Trace: [ 9396.175902] [ 9396.176860] ? show_regs.cold.9+0x22/0x2f [ 9396.180174] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9396.185735] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9396.194895] ? watchdog+0x40/0x40 [ 9396.199183] ? hrtimer_interrupt+0x128/0x2c0 [ 9396.203116] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9396.209505] ? apic_timer_interrupt+0xf/0x20 [ 9396.213342] [ 9396.215281] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9396.221267] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9396.224855] do_raw_spin_lock+0x177/0x190 [ 9396.235744] _raw_spin_lock+0x21/0x30 [ 9396.237924] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9396.242036] process_one_work+0x2c8/0x700 [ 9396.247918] worker_thread+0x296/0x6e0 [ 9396.251550] ? rescuer_thread+0x570/0x570 [ 9396.253879] kthread+0x1d1/0x200 [ 9396.256131] ? set_kthread_struct+0x70/0x70 [ 9396.258890] ret_from_fork+0x1f/0x30 [ 9404.053374] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 9404.055482] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 9404.062507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9404.062589] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9404.062598] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9404.062600] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9404.062615] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9404.062620] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9404.062625] RAX: 0000000000000e8c RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9404.062628] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9404.062630] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9404.062633] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9404.062635] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9404.062639] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9404.062642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9404.062645] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9404.062654] Call Trace: [ 9404.062660] [ 9404.062667] ? show_regs.cold.9+0x22/0x2f [ 9404.062676] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9404.062683] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9404.062688] ? watchdog+0x40/0x40 [ 9404.062693] ? hrtimer_interrupt+0x128/0x2c0 [ 9404.062697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9404.062704] ? apic_timer_interrupt+0xf/0x20 [ 9404.062709] [ 9404.062712] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9404.062716] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9404.076583] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9404.076666] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9404.076673] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9404.076676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9404.076691] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9404.076695] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9404.076700] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9404.076702] RDX: 00000000000074d5 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9404.076705] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9404.076707] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9404.076710] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9404.076713] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9404.076717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9404.076719] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9404.076728] Call Trace: [ 9404.076734] [ 9404.076740] ? show_regs.cold.9+0x22/0x2f [ 9404.076751] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9404.076758] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9404.076763] ? watchdog+0x40/0x40 [ 9404.076768] ? hrtimer_interrupt+0x128/0x2c0 [ 9404.076772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9404.076780] ? apic_timer_interrupt+0xf/0x20 [ 9404.076785] [ 9404.076788] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9404.076794] do_raw_spin_lock+0x177/0x190 [ 9404.076799] _raw_spin_lock+0x21/0x30 [ 9404.076805] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9404.077067] ? do_raw_spin_unlock+0x75/0x190 [ 9404.077073] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9404.079137] ? do_wait_intr+0xf0/0xf0 [ 9404.079148] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9404.080678] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9404.080895] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9404.081109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9404.081231] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9404.081274] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9404.081317] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9404.081346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9404.081433] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9404.081646] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9404.081676] lmv_intent_open+0x338/0x1150 [lmv] [ 9404.081708] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9404.081795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9404.081878] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9404.081901] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9404.081987] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9404.082072] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9404.082156] ? do_raw_spin_unlock+0x75/0x190 [ 9404.082164] ? _raw_spin_unlock+0x12/0x30 [ 9404.082172] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9404.082201] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9404.082232] ll_atomic_open+0x36a/0xd70 [lustre] [ 9404.082319] lookup_open+0x356/0xb40 [ 9404.082331] do_last+0x313/0xfc0 [ 9404.082336] ? nd_jump_root+0xe5/0x160 [ 9404.082341] ? path_init+0x437/0x520 [ 9404.082346] path_openat+0xf7/0x500 [ 9404.082351] ? do_raw_spin_unlock+0x75/0x190 [ 9404.082356] do_filp_open+0x99/0x140 [ 9404.082363] ? getname_flags+0x6e/0x330 [ 9404.082368] ? __check_object_size+0xff/0x256 [ 9404.082376] ? do_raw_spin_unlock+0x75/0x190 [ 9404.082380] ? _raw_spin_unlock+0x12/0x30 [ 9404.082385] do_sys_openat2+0x2b4/0x410 [ 9404.082390] do_sys_open+0x73/0xa0 [ 9404.082395] __x64_sys_openat+0x24/0x30 [ 9404.082398] do_syscall_64+0xc1/0x440 [ 9404.082405] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9404.082411] RIP: 0033:0x7fdc0afbf332 [ 9404.082419] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9404.082424] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9404.082429] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9404.082432] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9404.082435] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9404.082437] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9404.082440] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9404.647583] ? dump_stack+0xf6/0x10e [ 9404.649209] do_raw_spin_lock+0x177/0x190 [ 9404.652289] _raw_spin_lock+0x21/0x30 [ 9404.655963] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9404.660751] obd_mod_free+0x46/0x140 [ptlrpc] [ 9404.664431] mdc_commit_open+0x91/0xb0 [mdc] [ 9404.666538] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9404.673438] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9404.678143] after_reply+0xe2c/0x1780 [ptlrpc] [ 9404.680436] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9404.685825] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9404.688691] ? do_wait_intr+0xf0/0xf0 [ 9404.691107] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9404.693425] kthread+0x1d1/0x200 [ 9404.694592] ? set_kthread_struct+0x70/0x70 [ 9404.696603] ret_from_fork+0x1f/0x30 [ 9406.943031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7382s! [ 9406.947231] Showing busy workqueues and worker pools: [ 9406.958026] workqueue events: flags=0x0 [ 9406.961485] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9406.968169] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9406.975682] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9406.979686] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9406.987947] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9406.993904] pending: kfree_rcu_monitor [ 9406.997971] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9407.002562] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9407.011959] workqueue events_long: flags=0x0 [ 9407.013925] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9407.017739] pending: lu_objects_flush [obdclass] [ 9407.020874] workqueue events_unbound: flags=0x2 [ 9407.022501] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9407.027881] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9407.036806] workqueue events_power_efficient: flags=0x80 [ 9407.041928] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9407.049453] pending: neigh_periodic_work, check_lifetime [ 9407.053434] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9407.056956] pending: do_cache_clean, neigh_periodic_work [ 9407.061988] workqueue mm_percpu_wq: flags=0x8 [ 9407.065279] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9407.070042] pending: lru_add_drain_per_cpu [ 9407.073606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9407.081495] pending: vmstat_update, lru_add_drain_per_cpu [ 9407.084531] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9407.090042] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9407.095520] workqueue writeback: flags=0x4a [ 9407.096924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9407.101082] pending: wb_workfn, wb_workfn [ 9407.112470] workqueue xprtiod: flags=0xa [ 9407.115429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9407.123976] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9407.131340] workqueue ipv6_addrconf: flags=0x40008 [ 9407.138969] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9407.141827] pending: addrconf_verify_work [ 9407.146182] workqueue knbd0-recv: flags=0x1a [ 9407.154654] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9407.156927] in-flight: 655:recv_work [ 9407.172076] workqueue ptlrpc_pinger: flags=0x2 [ 9407.178570] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9407.182342] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9407.190559] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7382s workers=4 idle: 36396 38 11188 [ 9407.201468] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9397s workers=2 idle: 658 [ 9416.054037] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 9416.057156] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9416.076942] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9416.081867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9416.085813] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9416.087772] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9416.095732] RSP: 0018:ffffba8e41653ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9416.098775] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9416.101710] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9416.104534] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9416.107707] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9416.110659] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9416.113674] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9416.117005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9416.119519] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9416.122375] Call Trace: [ 9416.123523] [ 9416.124313] ? show_regs.cold.9+0x22/0x2f [ 9416.126046] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9416.128133] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9416.130032] ? watchdog+0x40/0x40 [ 9416.131550] ? hrtimer_interrupt+0x128/0x2c0 [ 9416.133317] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9416.135345] ? apic_timer_interrupt+0xf/0x20 [ 9416.137176] [ 9416.138082] ? native_safe_halt+0x2e/0x50 [ 9416.139826] kvm_wait+0xcd/0xe0 [ 9416.141155] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9416.143594] do_raw_spin_lock+0x177/0x190 [ 9416.145264] _raw_spin_lock+0x21/0x30 [ 9416.146931] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9416.149327] ? do_raw_spin_unlock+0x75/0x190 [ 9416.151112] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9416.154859] ? do_wait_intr+0xf0/0xf0 [ 9416.156493] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9416.159104] mdc_close+0x53b/0xd70 [mdc] [ 9416.160884] lmv_close+0x18d/0x490 [lmv] [ 9416.162548] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9416.165146] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9416.167320] ll_md_close+0x274/0x800 [lustre] [ 9416.169305] ll_file_release+0x128/0xc50 [lustre] [ 9416.171690] __fput+0x10d/0x440 [ 9416.172971] ____fput+0x12/0x20 [ 9416.174568] task_work_run+0xc6/0x110 [ 9416.176069] exit_to_usermode_loop+0x1dd/0x1f0 [ 9416.177954] do_syscall_64+0x3d6/0x440 [ 9416.179675] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9416.181867] RIP: 0033:0x7fdc0afbfcd8 [ 9416.183322] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9416.190944] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9416.194243] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9416.197459] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9416.200665] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9416.203705] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9416.206737] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9424.054423] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 9424.063337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9424.129067] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9424.137155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9424.147069] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9424.153564] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9424.157366] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9424.174807] RSP: 0018:ffffba8e40d9bdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9424.182441] RAX: 0000000000006584 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9424.189359] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9424.194867] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9424.200717] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9424.206926] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9424.212843] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9424.220683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9424.225885] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9424.231256] Call Trace: [ 9424.234568] [ 9424.235610] ? show_regs.cold.9+0x22/0x2f [ 9424.238223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9424.241337] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9424.242937] ? watchdog+0x40/0x40 [ 9424.244374] ? hrtimer_interrupt+0x128/0x2c0 [ 9424.249478] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9424.254675] ? apic_timer_interrupt+0xf/0x20 [ 9424.258953] [ 9424.259931] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9424.263504] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9424.266043] do_raw_spin_lock+0x177/0x190 [ 9424.268240] _raw_spin_lock+0x21/0x30 [ 9424.271201] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9424.276160] process_one_work+0x2c8/0x700 [ 9424.279082] worker_thread+0x296/0x6e0 [ 9424.284559] ? rescuer_thread+0x570/0x570 [ 9424.287679] kthread+0x1d1/0x200 [ 9424.290152] ? set_kthread_struct+0x70/0x70 [ 9424.293709] ret_from_fork+0x1f/0x30 [ 9432.050478] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2388] [ 9432.052253] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37519] [ 9432.056005] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9432.056086] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9432.056094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9432.056097] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9432.056112] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 9432.056117] RSP: 0018:ffffba8e41bf7bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9432.056121] RAX: 000000000000341f RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9432.056124] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe642034154 [ 9432.056126] RBP: ffff9fe642034140 R08: 0000000000000000 R09: ffffffff93537fe0 [ 9432.056128] R10: ffffffff92008b80 R11: ffffba8e41bf78f0 R12: 0000000000040000 [ 9432.056131] R13: ffff9fe6421b4140 R14: 000000000001e480 R15: ffff9fe64201e480 [ 9432.056135] FS: 0000000000000000(0000) GS:ffff9fe642000000(0000) knlGS:0000000000000000 [ 9432.056138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9432.056141] CR2: 00005591c8ceb140 CR3: 0000000105016003 CR4: 0000000000170ef0 [ 9432.060715] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9432.060801] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9432.060809] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9432.060811] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9432.060825] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9432.060829] RSP: 0018:ffffba8e415e3500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9432.060834] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9432.060836] RDX: 0000000000003b8d RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9432.060839] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9432.060841] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9432.060844] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9432.060847] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9432.060850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9432.060852] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9432.060860] Call Trace: [ 9432.060865] [ 9432.060870] ? show_regs.cold.9+0x22/0x2f [ 9432.061134] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9432.061141] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9432.061145] ? watchdog+0x40/0x40 [ 9432.061149] ? hrtimer_interrupt+0x128/0x2c0 [ 9432.061153] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9432.061161] ? apic_timer_interrupt+0xf/0x20 [ 9432.061166] [ 9432.061170] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9432.061175] do_raw_spin_lock+0x177/0x190 [ 9432.061180] _raw_spin_lock+0x21/0x30 [ 9432.061185] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9432.061666] ? do_raw_spin_unlock+0x75/0x190 [ 9432.061673] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9432.061881] ? do_wait_intr+0xf0/0xf0 [ 9432.061890] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9432.092804] Call Trace: [ 9432.092816] [ 9432.092821] ? show_regs.cold.9+0x22/0x2f [ 9432.092834] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9432.092840] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9432.092844] ? watchdog+0x40/0x40 [ 9432.092848] ? hrtimer_interrupt+0x128/0x2c0 [ 9432.092852] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9432.092861] ? apic_timer_interrupt+0xf/0x20 [ 9432.092865] [ 9432.092868] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9432.092874] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9432.092878] ? dump_stack+0xf6/0x10e [ 9432.092883] do_raw_spin_lock+0x177/0x190 [ 9432.092887] _raw_spin_lock+0x21/0x30 [ 9432.092894] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9432.094602] obd_mod_free+0x46/0x140 [ptlrpc] [ 9432.094817] mdc_commit_open+0x91/0xb0 [mdc] [ 9432.094859] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9432.097890] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9432.098102] after_reply+0xe2c/0x1780 [ptlrpc] [ 9432.098301] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9432.098664] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9432.098874] ? do_wait_intr+0xf0/0xf0 [ 9432.098886] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9432.099402] kthread+0x1d1/0x200 [ 9432.099409] ? set_kthread_struct+0x70/0x70 [ 9432.099413] ret_from_fork+0x1f/0x30 [ 9432.297161] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9432.299691] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9432.301914] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9432.303802] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9432.305480] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9432.307023] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9432.308818] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9432.310760] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9432.313289] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9432.315009] lmv_intent_open+0x338/0x1150 [lmv] [ 9432.316666] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9432.318479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9432.320239] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9432.321820] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9432.323441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9432.325283] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9432.326867] ? do_raw_spin_unlock+0x75/0x190 [ 9432.328599] ? _raw_spin_unlock+0x12/0x30 [ 9432.329827] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9432.331644] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9432.333496] ll_atomic_open+0x36a/0xd70 [lustre] [ 9432.335268] lookup_open+0x356/0xb40 [ 9432.336901] do_last+0x313/0xfc0 [ 9432.338185] ? nd_jump_root+0xe5/0x160 [ 9432.339775] ? path_init+0x437/0x520 [ 9432.341256] path_openat+0xf7/0x500 [ 9432.342519] ? do_raw_spin_unlock+0x75/0x190 [ 9432.344477] do_filp_open+0x99/0x140 [ 9432.345736] ? getname_flags+0x6e/0x330 [ 9432.347061] ? __check_object_size+0xff/0x256 [ 9432.348794] ? do_raw_spin_unlock+0x75/0x190 [ 9432.350527] ? _raw_spin_unlock+0x12/0x30 [ 9432.351716] do_sys_openat2+0x2b4/0x410 [ 9432.353273] do_sys_open+0x73/0xa0 [ 9432.354255] __x64_sys_openat+0x24/0x30 [ 9432.355690] do_syscall_64+0xc1/0x440 [ 9432.356837] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9432.358737] RIP: 0033:0x7fdc0afbf332 [ 9432.359963] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9432.366346] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9432.368701] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9432.371549] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9432.374907] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9432.378500] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9432.380864] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9437.663037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7412s! [ 9437.666911] Showing busy workqueues and worker pools: [ 9437.669729] workqueue events: flags=0x0 [ 9437.671914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9437.676108] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9437.679698] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9437.682890] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9437.688352] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9437.692232] pending: kfree_rcu_monitor [ 9437.694765] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9437.698013] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9437.703918] workqueue events_long: flags=0x0 [ 9437.706175] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9437.709780] pending: lu_objects_flush [obdclass] [ 9437.711703] workqueue events_unbound: flags=0x2 [ 9437.715053] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9437.720535] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9437.725720] workqueue events_power_efficient: flags=0x80 [ 9437.728095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9437.731551] pending: neigh_periodic_work, check_lifetime [ 9437.734117] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9437.737340] pending: do_cache_clean, neigh_periodic_work [ 9437.739798] workqueue mm_percpu_wq: flags=0x8 [ 9437.742041] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9437.744894] pending: lru_add_drain_per_cpu [ 9437.746627] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9437.749340] pending: vmstat_update, lru_add_drain_per_cpu [ 9437.752112] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9437.755051] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9437.758102] workqueue writeback: flags=0x4a [ 9437.760012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9437.763010] pending: wb_workfn, wb_workfn [ 9437.764850] workqueue xprtiod: flags=0xa [ 9437.766613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9437.769370] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9437.773332] workqueue ipv6_addrconf: flags=0x40008 [ 9437.775619] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9437.778718] pending: addrconf_verify_work [ 9437.780871] workqueue knbd0-recv: flags=0x1a [ 9437.785693] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9437.788421] in-flight: 655:recv_work [ 9437.791973] workqueue ptlrpc_pinger: flags=0x2 [ 9437.795219] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9437.798163] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9437.805737] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7412s workers=4 idle: 36396 38 11188 [ 9437.810821] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9428s workers=2 idle: 658 [ 9444.056870] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 9444.065106] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9444.112083] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9444.118224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9444.122885] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9444.127916] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9444.141834] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9444.148603] RAX: 0000000000005003 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9444.152872] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9444.158388] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9444.163622] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9444.172440] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9444.177715] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9444.181666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9444.185009] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9444.191553] Call Trace: [ 9444.193284] [ 9444.194021] ? show_regs.cold.9+0x22/0x2f [ 9444.196649] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9444.200222] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9444.203056] ? watchdog+0x40/0x40 [ 9444.205432] ? hrtimer_interrupt+0x128/0x2c0 [ 9444.207186] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9444.210435] ? apic_timer_interrupt+0xf/0x20 [ 9444.211598] [ 9444.212098] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9444.214493] do_raw_spin_lock+0x177/0x190 [ 9444.217409] _raw_spin_lock+0x21/0x30 [ 9444.221952] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9444.224046] ? do_raw_spin_unlock+0x75/0x190 [ 9444.227381] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9444.230387] ? do_wait_intr+0xf0/0xf0 [ 9444.232383] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9444.234165] mdc_close+0x53b/0xd70 [mdc] [ 9444.235411] lmv_close+0x18d/0x490 [lmv] [ 9444.237043] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9444.240388] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9444.245120] ll_md_close+0x274/0x800 [lustre] [ 9444.250139] ll_file_release+0x128/0xc50 [lustre] [ 9444.252210] __fput+0x10d/0x440 [ 9444.254227] ____fput+0x12/0x20 [ 9444.256277] task_work_run+0xc6/0x110 [ 9444.258414] exit_to_usermode_loop+0x1dd/0x1f0 [ 9444.260544] do_syscall_64+0x3d6/0x440 [ 9444.262649] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9444.265095] RIP: 0033:0x7fdc0afbfcd8 [ 9444.266516] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9444.278603] RSP: 002b:00007ffea5711968 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9444.283475] RAX: 0000000000000000 RBX: 000000000000b750 RCX: 00007fdc0afbfcd8 [ 9444.287013] RDX: 0000000000000038 RSI: 00007ffea5711dd0 RDI: 0000000000000003 [ 9444.292685] RBP: 00007ffea5711dd0 R08: 00007fdc0b262234 R09: 00007fdc0b262260 [ 9444.295757] R10: 00007ffea5711712 R11: 0000000000000246 R12: 0000000000000003 [ 9444.300947] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9452.051015] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:1:36] [ 9452.059928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9452.096445] CPU: 1 PID: 36 Comm: kworker/u8:1 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9452.103483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9452.111488] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9452.119500] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9452.122680] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9452.133970] RSP: 0018:ffffba8e40d9bdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9452.138393] RAX: 0000000000000001 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9452.143703] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9fe6420b4154 [ 9452.154964] RBP: ffff9fe6420b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9452.158713] R10: 000000000469f604 R11: ffff9fe6159d0000 R12: 0000000000080000 [ 9452.166872] R13: ffff9fe642034140 R14: 000000000001e480 R15: ffff9fe64209e480 [ 9452.173565] FS: 0000000000000000(0000) GS:ffff9fe642080000(0000) knlGS:0000000000000000 [ 9452.179517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9452.184534] CR2: 00007f665ddb0000 CR3: 0000000105016006 CR4: 0000000000170ee0 [ 9452.190369] Call Trace: [ 9452.192843] [ 9452.194195] ? show_regs.cold.9+0x22/0x2f [ 9452.196345] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9452.197947] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9452.200882] ? watchdog+0x40/0x40 [ 9452.203148] ? hrtimer_interrupt+0x128/0x2c0 [ 9452.204406] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9452.209148] ? apic_timer_interrupt+0xf/0x20 [ 9452.211057] [ 9452.212324] ? native_safe_halt+0x2e/0x50 [ 9452.214142] kvm_wait+0xcd/0xe0 [ 9452.215419] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9452.217306] do_raw_spin_lock+0x177/0x190 [ 9452.218854] _raw_spin_lock+0x21/0x30 [ 9452.220634] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9452.223143] process_one_work+0x2c8/0x700 [ 9452.224815] worker_thread+0x296/0x6e0 [ 9452.228517] ? rescuer_thread+0x570/0x570 [ 9452.231589] kthread+0x1d1/0x200 [ 9452.233745] ? set_kthread_struct+0x70/0x70 [ 9452.238352] ret_from_fork+0x1f/0x30 [ 9459.519444] rcu: INFO: rcu_sched self-detected stall on CPU [ 9459.523155] rcu: 0-...!: (4666084 ticks this GP) idle=882/1/0x4000000000000002 softirq=130586/130586 fqs=869 [ 9459.533643] (t=7440176 jiffies g=270761 q=4494) [ 9459.535390] rcu: rcu_sched kthread starved for 7435108 jiffies! g270761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 9459.540651] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9459.544599] rcu: RCU grace-period kthread stack dump: [ 9459.547760] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9459.552699] Call Trace: [ 9459.555339] __schedule+0x351/0xcb0 [ 9459.557244] schedule+0xc0/0x180 [ 9459.558505] schedule_timeout+0xb4/0x190 [ 9459.560874] ? do_raw_spin_unlock+0x75/0x190 [ 9459.564233] ? __next_timer_interrupt+0x160/0x160 [ 9459.568372] ? prepare_to_swait_event+0xf3/0x140 [ 9459.570639] rcu_gp_kthread+0x8dd/0x15b0 [ 9459.576479] ? force_qs_rnp+0x2c0/0x2c0 [ 9459.579606] kthread+0x1d1/0x200 [ 9459.581225] ? set_kthread_struct+0x70/0x70 [ 9459.584079] ret_from_fork+0x1f/0x30 [ 9459.588276] rcu: Stack dump where RCU GP kthread last ran: [ 9459.591665] NMI backtrace for cpu 0 [ 9459.593798] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9459.611748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9459.624589] Call Trace: [ 9459.630515] [ 9459.638358] ? dump_stack+0xbb/0x10e [ 9459.639672] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9459.646930] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9459.653427] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9459.659360] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9459.663375] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9459.670177] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 9459.673000] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 9459.675283] ? update_process_times+0x6d/0xc0 [ 9459.676844] ? tick_sched_handle.isra.18+0x27/0x90 [ 9459.682747] ? tick_sched_timer+0x53/0xb0 [ 9459.684658] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9459.686768] ? tick_sched_handle.isra.18+0x90/0x90 [ 9459.689233] ? hrtimer_interrupt+0x128/0x2c0 [ 9459.693651] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9459.695600] ? apic_timer_interrupt+0xf/0x20 [ 9459.697537] [ 9459.698599] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9459.702464] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9459.705784] ? dump_stack+0xf6/0x10e [ 9459.708427] ? do_raw_spin_lock+0x177/0x190 [ 9459.710402] ? _raw_spin_lock+0x21/0x30 [ 9459.712165] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9459.716466] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9459.719215] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9459.721431] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9459.725239] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9459.729246] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9459.731077] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9459.740849] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9459.742204] ? do_wait_intr+0xf0/0xf0 [ 9459.743127] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9459.755819] ? kthread+0x1d1/0x200 [ 9459.756707] ? set_kthread_struct+0x70/0x70 [ 9459.765684] ? ret_from_fork+0x1f/0x30 [ 9459.766904] NMI backtrace for cpu 0 [ 9459.772132] CPU: 0 PID: 2388 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9459.775675] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9459.778618] Call Trace: [ 9459.780402] [ 9459.780990] ? dump_stack+0xbb/0x10e [ 9459.781943] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9459.786180] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9459.791095] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9459.796368] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9459.802421] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9459.804257] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9459.807633] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9459.811590] ? update_process_times+0x6d/0xc0 [ 9459.816068] ? tick_sched_handle.isra.18+0x27/0x90 [ 9459.818399] ? tick_sched_timer+0x53/0xb0 [ 9459.821026] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9459.824235] ? tick_sched_handle.isra.18+0x90/0x90 [ 9459.828984] ? hrtimer_interrupt+0x128/0x2c0 [ 9459.831573] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9459.834724] ? apic_timer_interrupt+0xf/0x20 [ 9459.839308] [ 9459.841480] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9459.845774] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9459.851916] ? dump_stack+0xf6/0x10e [ 9459.853299] ? do_raw_spin_lock+0x177/0x190 [ 9459.854734] ? _raw_spin_lock+0x21/0x30 [ 9459.856979] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9459.863066] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9459.871283] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9459.873491] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9459.881559] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9459.884265] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9459.886624] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9459.890974] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9459.893968] ? do_wait_intr+0xf0/0xf0 [ 9459.896878] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9459.901880] ? kthread+0x1d1/0x200 [ 9459.905743] ? set_kthread_struct+0x70/0x70 [ 9459.909387] ? ret_from_fork+0x1f/0x30 [ 9460.052069] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37519] [ 9460.055338] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9460.074652] CPU: 2 PID: 37519 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9460.078700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9460.083369] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9460.085456] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9460.095062] RSP: 0018:ffffba8e415e34f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9460.100898] RAX: 0000000000000003 RBX: ffff9fe615422ae0 RCX: ffff9fe64211e480 [ 9460.104824] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9fe615422ae0 [ 9460.107948] RBP: ffff9fe642134140 R08: 0000000000000008 R09: 00000000000000b8 [ 9460.113093] R10: 00000000046884c5 R11: ffff9fe614233000 R12: 00000000000c0000 [ 9460.116912] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9460.121765] FS: 00007fdc0b690740(0000) GS:ffff9fe642100000(0000) knlGS:0000000000000000 [ 9460.125728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9460.128642] CR2: 000056124e9d4658 CR3: 0000000113a29001 CR4: 0000000000170ee0 [ 9460.131995] Call Trace: [ 9460.133448] [ 9460.134364] ? show_regs.cold.9+0x22/0x2f [ 9460.136412] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9460.137846] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9460.139200] ? watchdog+0x40/0x40 [ 9460.140321] ? hrtimer_interrupt+0x128/0x2c0 [ 9460.142944] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9460.146230] ? apic_timer_interrupt+0xf/0x20 [ 9460.149560] [ 9460.150569] ? native_safe_halt+0x2e/0x50 [ 9460.152223] kvm_wait+0xcd/0xe0 [ 9460.154165] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9460.156578] do_raw_spin_lock+0x177/0x190 [ 9460.158799] _raw_spin_lock+0x21/0x30 [ 9460.160048] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9460.163231] ? do_raw_spin_unlock+0x75/0x190 [ 9460.165288] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9460.168201] ? do_wait_intr+0xf0/0xf0 [ 9460.169931] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9460.174182] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9460.176959] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9460.181753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9460.185073] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9460.187817] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9460.190995] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9460.192865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9460.194907] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9460.197783] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9460.199813] lmv_intent_open+0x338/0x1150 [lmv] [ 9460.201745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9460.205762] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9460.208039] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9460.211392] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9460.216881] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9460.220552] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9460.223690] ? do_raw_spin_unlock+0x75/0x190 [ 9460.225496] ? _raw_spin_unlock+0x12/0x30 [ 9460.226980] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9460.228660] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9460.231619] ll_atomic_open+0x36a/0xd70 [lustre] [ 9460.233532] lookup_open+0x356/0xb40 [ 9460.234558] do_last+0x313/0xfc0 [ 9460.236118] ? nd_jump_root+0xe5/0x160 [ 9460.236934] ? path_init+0x437/0x520 [ 9460.237808] path_openat+0xf7/0x500 [ 9460.240669] ? do_raw_spin_unlock+0x75/0x190 [ 9460.242001] do_filp_open+0x99/0x140 [ 9460.243036] ? getname_flags+0x6e/0x330 [ 9460.244154] ? __check_object_size+0xff/0x256 [ 9460.245407] ? do_raw_spin_unlock+0x75/0x190 [ 9460.246558] ? _raw_spin_unlock+0x12/0x30 [ 9460.247664] do_sys_openat2+0x2b4/0x410 [ 9460.248750] do_sys_open+0x73/0xa0 [ 9460.249715] __x64_sys_openat+0x24/0x30 [ 9460.250969] do_syscall_64+0xc1/0x440 [ 9460.252640] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9460.255038] RIP: 0033:0x7fdc0afbf332 [ 9460.255920] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9460.262519] RSP: 002b:00007ffea5711900 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9460.265758] RAX: ffffffffffffffda RBX: 0000000000001768 RCX: 00007fdc0afbf332 [ 9460.268451] RDX: 0000000000000042 RSI: 00007ffea57119d0 RDI: 00000000ffffff9c [ 9460.270683] RBP: 00007ffea5711dd0 R08: 0000000000000000 R09: 0000000000000000 [ 9460.273109] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffea57119d0 [ 9460.275962] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 9467.871555] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7442s! [ 9467.877835] Showing busy workqueues and worker pools: [ 9467.890088] workqueue events: flags=0x0 [ 9467.892132] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9467.894079] pending: cpu_latency_work [ptlrpc], push_to_pool [ 9467.896629] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9467.901831] pending: cpu_latency_work [ptlrpc], osc_grant_work_handler [osc], key_garbage_collector [ 9467.907265] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9467.910418] pending: kfree_rcu_monitor [ 9467.913522] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9467.917566] pending: vmstat_shepherd, nfs4_renew_state [nfsv4], ldlm_pools_recalc_task [ptlrpc], sec_gc_main [ptlrpc] [ 9467.930508] workqueue events_long: flags=0x0 [ 9467.935069] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9467.939655] pending: lu_objects_flush [obdclass] [ 9467.942555] workqueue events_unbound: flags=0x2 [ 9467.944596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9467.947506] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9467.953320] workqueue events_power_efficient: flags=0x80 [ 9467.956463] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9467.958838] pending: neigh_periodic_work, check_lifetime [ 9467.960860] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9467.963510] pending: do_cache_clean, neigh_periodic_work [ 9467.965435] workqueue mm_percpu_wq: flags=0x8 [ 9467.967124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9467.969516] pending: lru_add_drain_per_cpu [ 9467.972218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9467.975179] pending: vmstat_update, lru_add_drain_per_cpu [ 9467.977966] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9467.980978] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9467.982900] workqueue writeback: flags=0x4a [ 9467.984483] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9467.987616] pending: wb_workfn, wb_workfn [ 9467.989548] workqueue xprtiod: flags=0xa [ 9467.991180] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9467.993993] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9467.997353] workqueue ipv6_addrconf: flags=0x40008 [ 9467.999499] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9468.002685] pending: addrconf_verify_work [ 9468.007142] workqueue knbd0-recv: flags=0x1a [ 9468.009480] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9468.021065] in-flight: 655:recv_work [ 9468.023692] workqueue ptlrpc_pinger: flags=0x2 [ 9468.028378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9468.032918] in-flight: 36:ptlrpc_pinger_main [ptlrpc] [ 9468.036125] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7443s workers=4 idle: 36396 38 11188 [ 9468.040276] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9458s workers=2 idle: 658 [ 9472.066157] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:37520] [ 9472.079587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9472.131859] CPU: 3 PID: 37520 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9472.146647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9472.158935] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9472.160788] Code: 4c 03 2c c5 80 58 e8 91 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9472.193979] RSP: 0018:ffffba8e41653af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9472.199482] RAX: 0000000000006e85 RBX: ffff9fe615422ae0 RCX: 0000000000000001 [ 9472.211988] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9fe6421b4154 [ 9472.221331] RBP: ffff9fe6421b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9472.229154] R10: 000000000468bfd0 R11: ffff9fe6158c8000 R12: 0000000000100000 [ 9472.241580] R13: ffff9fe642134140 R14: 000000000001e480 R15: ffff9fe64219e480 [ 9472.250418] FS: 00007fdc0b690740(0000) GS:ffff9fe642180000(0000) knlGS:0000000000000000 [ 9472.260173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9472.265056] CR2: 000056124e9c4018 CR3: 0000000118e6f004 CR4: 0000000000170ee0 [ 9472.278800] Call Trace: [ 9472.279719] [ 9472.283319] ? show_regs.cold.9+0x22/0x2f [ 9472.285149] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9472.290732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9472.292961] ? watchdog+0x40/0x40 [ 9472.296017] ? hrtimer_interrupt+0x128/0x2c0 [ 9472.300561] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9472.313428] ? apic_timer_interrupt+0xf/0x20 [ 9472.314970] [ 9472.315754] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9472.332934] do_raw_spin_lock+0x177/0x190 [ 9472.336848] _raw_spin_lock+0x21/0x30 [ 9472.339454] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9472.343335] ? do_raw_spin_unlock+0x75/0x190 [ 9472.344943] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9472.347724] ? do_wait_intr+0xf0/0xf0 [ 9472.357081] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9472.3629